Two identical docs-generator runs disagree on 77% of all.json

Context

Found while producing the real-tree equivalence receipt @neo-gpt required in RA-2 of review PRR_kwDODSospM8AAAABKdTQvA on PR #17496. The receipt could not be produced for one of the three outputs, and the reason is a pre-existing property of the generator rather than anything that PR changes.

Live latest-open sweep: checked latest 6 open issues at 2026-08-21T19:5x UTC; gh search issues for docs generator deterministic OR reproducible OR all.json returned zero. No equivalent exists.

The Problem

npm run generate-docs-json is not reproducible. Running it twice with identical source and identical code, in the same working directory, produces materially different output.

Measured on origin/dev, two consecutive runs:

For structure.json the variance is confined to identity-free fields: the (className, name, path, isLeaf, singleton, srcPath) tuple set is identical across runs, and so is the parent graph once expressed by identity rather than by the volatile id. So that output is semantically stable and only its numbering is not.

all.json is worse. Normalizing array ordering does not converge it, and the fields that differ between two identical runs include mixes, comment, memberof, augments, $longname and meta — content, not just ordering.

Why it matters beyond tidiness

It disables a whole class of evidence. Any before/after comparison over all.json is uninterpretable: on this measurement the same-code control (15,189 differing records) is the same order of magnitude as a real change (15,926), and the second control pairing was larger than the subject pairing (16,064). A reviewer asking "did this change alter the generated docs?" cannot be answered from the artifact, and — worse — a comparison can be run, will produce a confident-looking diff, and will mean nothing. That is the failure direction: not an error, a plausible answer.

It also means the committed docs output cannot be regenerated and diffed to verify it is current, and any future CI check of the form "regenerate and assert no drift" would be permanently red for reasons unrelated to drift.

The Architectural Reality

• buildScripts/docs/generateDocsJson.mjs — writes docs/output/{all,class-hierarchy,structure}.json
• buildScripts/docs/docletPipeline/ — runner.mjs batches the parse; sorter.mjs, transformer.mjs
• The structure.json id/parentId numbering is assigned during tree construction

The structure.json symptom (stable identities, unstable numbering) and the all.json symptom (unstable content) are probably two different causes and should be diagnosed separately. Ordering variance across a batched/concurrent parse would explain the first; the second needs investigation before a cause is asserted — I have not diagnosed it and am not guessing here.

Acceptance Criteria

• Two consecutive generate-docs-json runs on an unchanged tree produce byte-identical structure.json and class-hierarchy.json
• Either the same holds for all.json, or its residual variance is characterised, justified, and documented as intentional with a named stable projection consumers can compare on
• The structure.json id assignment is deterministic for a given source set — asserted by a test that generates twice and compares, not by inspection
• The determinism check runs in CI, so this cannot silently return
• The diagnosis records why each output varied; a fix that makes the symptom stop without naming the cause does not satisfy this

Out of Scope

• Any change to what the docs contain — this is about the same input yielding the same output.
• PR #17496's namespace-walker fix, which this blocks nothing in; its equivalence receipt is delivered on the two reproducible outputs plus the identity/parent-graph projection of structure.json.

Avoided Traps

Assuming the cause is ordering. It explains structure.json and is refuted for all.json — normalizing array order does not converge two identical runs. Fixing sort stability and declaring victory would leave 77% of the largest artifact still varying.

Treating the committed docs/output as a baseline. It cannot be reproduced, so it is not a reference; comparing against it measures nothing.

Related

• PR #17496 / #17494 — where this surfaced, via RA-2's real-tree receipt requirement

Retrieval Hint: generate-docs-json nondeterministic reproducible all.json structure.json id assignment

Origin Session ID: 752da6ac-a6c3-447f-8847-1da4ce49deb8

Decision Record impact: none — build-script determinism, no ADR authority touched. Structure-map gate: N/A, no ai/ surface and no file placement.