🎯 Summary

The local pull-request corpus holds 2,015 _index.json entries naming paths that do not exist and 27 PR ids owning two byte-divergent artifacts each. It has been like this for months, and every pass reported success the entire time.

Both symptoms are one cause: every surface that placed a PR ranked it against a partial view of its bucket. metadata.pulls is rebuilt from each run's delta fetch, so a planner consulting it sees a fraction of a sealed bucket. An ordinal derived from a partial collection is not a smaller truth — it is a different number. The PR resolves to a chunk the complete ordering would never choose, the write lands beside the existing artifact instead of on it, and the two renderings diverge from that moment. The old copy survives because the unlink is gated on cachedPull?.path, and a cache miss makes that null.

This matters past tidiness because the corpus feeds the Knowledge Base. The ticket's own AC names the stakes better than I can: duplicate projection "masquerading as distinct evidence." Our verify-before-assert substrate can currently hand a maintainer two divergent copies of one PR and present them as independent corroboration.

Resolves #15130

Refs #13001, #13200 (the incomplete persistence boundary), #15088 (discovered by, non-blocking)

Evidence: L3 (the validator and both repair passes executed against a byte-copy of the live 4,661-artifact corpus — 2,015 stale → 0, partition verified; 560 specs green; tracked corpus asserted unmodified after a full suite run) → L4 required (AC5 and AC7 name post-repair counts in the TRACKED corpus, which is written only by a sync_all run on dev — an operator/agent action that commits generated content, deliberately not taken from this branch at night and unreviewed). Residual: AC5, AC7 [#15130].

Correction to an earlier draft of this line, because the name is a trap. It read "only the data-sync pipeline may write — agents do not run it." That is false, and two different things share the name. resources/content is written by sync_all (MCP → syncAllOnDevOnly → SyncService.runFullSync), which is agent-invoked and dev-guarded — it is the documented exception to no-direct-commit, not a CI job. The GitHub workflow called "Data Sync Pipeline" reads resources/content to publish it to neomjs/pages and commits devindex/portal artifacts; it never writes the corpus. So the residual is real but its reason was wrong: the repair lands on the next sync_all, run by a person or an agent, not on a CI schedule.

🔬 Evidence — I re-censused before trusting the ticket

@neo-gpt's forensics are three days old, so I ran them again rather than inherit them. They reproduce exactly:

Both of his named falsifiers land: the first stale entry my probe returns is pr-9537, the first duplicate is 10124. And the duplicate's shape confirms the diagnosis rather than merely its count:

archive/pulls/v13.0.0/chunk-1/pr-10124.md   1857 B
archive/pulls/v13.0.0/chunk-2/pr-10124.md   1790 B

Same sealed version, two chunk ordinals, divergent bytes. Not a move that forgot an upsert — a second placement derived from membership the planner could not see.

🧭 The changes

1 · The complete corpus, and a verdict over it. buildContentInventory scans both tiers — files outlive every cache describing them, so the corpus is the only complete membership that exists. It keys ids to an array, because "one artifact per id" is the invariant under test, not an assumption the scanner may make: a Map<id, entry> would drop the second copy and report a clean corpus, which is exactly the blindness that let this accumulate.

2 · A move and its index upsert are one mutation set. reconcileClosedPullRequestLocations renamed files and never touched the index. A rename that does not carry its entry does not relocate a file — it hides it: the artifact is intact, the lookup points at nothing, nothing complains. "The next sync rebuilds the index" was the standing assumption and is false for exactly this set: syncPullRequests upserts only its own delta, and a marooned backlog is the set the delta never names.

3 · An archived PR keeps the artifact it owns. Archive ordinals are fixed when a bucket is cut, so recomputing one at refresh time answers a different question than the one that placed the file. Reusing the occupied location makes a refresh overwrite — a rival copy becomes unreachable by construction rather than merely unlikely. Placement now plans against the corpus; a PR owning two artifacts is refused, not guessed at, because nothing on disk says which is current and writing to either canonicalises a guess.

4 · The index repair. Preventing new drift does not remove old drift, and nothing else ever would have — the 2,015 name files already archived, so the relocate pass never revisits them and the delta never fetches them. They were unreachable by every mechanism expected to heal them. The index is a projection of the corpus, so reconcilePullRequestIndex recomputes it from disk, every sync, idempotently.

5 · The duplicate restore. The 27 cannot be resolved locally, so GitHub decides. Fetch before unlink — the copies are the only local record, so deleting first would turn a network blip into a corpus simply missing the PR; a repair that can lose data on a bad connection is not a repair. A PR absent from GitHub is refused rather than cleaned up, since its copies are then the only record left. Both repairs reach the corpus through the normal sync, which is what the ticket asked for and why it forbade a migration script: a script clears the rows once and leaves the class alive.

Also fixed, both latent and neither in the ticket:

• The end-of-run metadata loop assumed every fetched PR produced a path and sits outside the per-PR catch — so any PR failing to sync, for any reason, long before this change, threw on path.resolve(root, undefined) and took down the whole run's metadata and index write.
• refetchPullsByNumber — the recovery primitive — planned from a single PR, the narrowest view in the syncer. A tool that manufactures the drift it exists to remove is worse than no tool, because it runs precisely when the corpus is already suspect.

🔁 Review cycle 1 — @neo-gpt-emmy's four coupled boundaries, all real

Her review found four groups. Every one was a real defect, and two were the same shape twice.

RA-2 — duplicate repair was destructive under every failure point but the one I'd thought of. I wrote a commit message about "fetch before unlink: a repair that can lose data on a bad connection is not a repair" — while the canonical rendering sat in MEMORY as both copies were deleted. Planning, path resolution (my own ambiguity refusal, firing from inside that window), mkdir, the write: any throw, or a crash needing no exception at all, left the corpus missing the PR and logged it as a soft skip. "Fetched" is not "durable"; only a file on disk is. Now: plan → temp write → atomic rename → then remove stale copies, never the one just written. Separately, reconcileClosedPullRequestLocations called fs.rename() with no destination check — and rename silently replaces, so it could resolve a duplicate by deletion, destroying the archived copy the repair arbitrates from, in the pass that runs before the refusal to do exactly that.

RA-1 — complete membership was only half complete. #planBuckets inherited corpus membership for versioned buckets only, so the active collection still ranked PRs against the delta — the same partial-ordinal defect one tier over, invisible because the archive half looked fixed. And parseContentPath hardcoded chunk- / /^v\d/ while the config carries archiveChunkPrefix / versionDirectoryPrefix: the hardcodes agree with the shipped defaults, which is why nothing was red. Under an override the forward direction builds slice-3/ while the inverse only recognises chunk-3/ — two halves of one contract disagreeing silently.

RA-3 — the lane could measure the corpus as broken and commit it anyway. Every pass reported its own outcome and degraded softly on its own terms; nothing downstream asked. One verdict now runs after placement, restoration and projection are final, and aborts before the metadata save, the derive, and the auto-push. A failed run is retryable; a generated commit is what every consumer then reads as truth. The index also became an actual projection — it now removes: a row for an ambiguous id blesses one of two divergent copies as canonical, silently, because the path is real.

RA-4 — the consumer boundary. PullRequestSource keys chunks by PR identity while walking files, so two artifacts emitted two chunks under the same pr-<id>#body name with divergent content. It now refuses. Ids are normalised to strings because they arrive as Number from the index and String from the filename fallback — compared raw, 10124 !== '10124' and the guard silently never fires.

Twice she found the same shape: the branch I thought of fixed, the invariant's other half open. That is the tell from #14153 — a ticket I re-opened this same evening, for this. My specs could not catch it because each exercised the half I had already fixed.

Test Evidence

560 specs green across github-workflow + knowledge-base (no collateral damage). The tracked corpus is unmodified after a full suite run — asserted, not assumed, because a green suite is exactly what it looked like while it was rewriting 4,180 lines of it.

One pre-existing failure is NOT mine: PullRequestService.spec.mjs:399 (getPullRequestDiff file filter) fails on this branch and on clean origin/dev with these changes absent — verified in a detached worktree at f3b36f627b. Zero coupling: that service references none of these surfaces.

The repair, measured against a throwaway copy of the live corpus — tracked files never touched:

BEFORE  stale indexed paths : 2015
repair  reindexed 2015 | already correct 2592 | skipped ambiguous 27
AFTER   stale indexed paths : 0

2015 + 2592 + 27 = 4,634 — exactly the unique-id count. The pass partitions the corpus with every id accounted for and none counted twice, and it correctly skips the 27 rather than indexing one copy of each: an entry naming one of two divergent artifacts would bless it as canonical by implication. Those are handled by the restore, which is the only step allowed to decide, and it decides by asking GitHub.

Red-then-green, proven by stashing the fix, not asserted: the lookup assertion reads pulls/chunk-1/pr-13001.md where the file is at archive/pulls/v13.0.0/chunk-1/ — the production defect reproduced. 4 fail without it.

The specs assert the defect BEFORE any counter this change introduces. Ordered the other way they tripped on the new field first and never evaluated the defect at all — red for a bookkeeping reason, which is a test that cannot fail for its stated purpose. My first draft did exactly that; the ordering is the fix.

Two instruments agree: the tested validator independently reproduces an ad-hoc probe written against a different shape — 2,015 / 27 / 4,634, all matching.

Every detection case is paired with a positive control proving the same probe returns non-zero when the condition is present, and the clean-corpus case proves the verdict can reach PASS. A validator that cannot fail and one that cannot pass are equally worthless. Three of the first probes I wrote for this lane returned clean zeros from blind instruments (wrong container shape, wrong extension, wrong filename prefix); any one, trusted, closes this ticket as already-fixed. The controls are why the surviving numbers mean anything.

Post-Merge Validation

⚠️ The repair does not run by itself. Someone must run it, on dev, after this merges.

Stated loudly because I nearly shipped this lane with the opposite implication. Verified:

• No workflow runs the content sync. The GitHub workflow named "Data Sync Pipeline" only READS resources/content to publish it to neomjs/pages; it commits devindex/portal artifacts and never writes the corpus.
• syncOnStartup defaults to false — an MCP server boot does not trigger it.
• The only entrypoints are npm run ai:sync-github-workflow (dev-branch-guarded CLI, takes a heavy-maintenance lease) and the sync_all MCP tool — both manual, both dev-only.

So five commits of repair merge and then sit inert until a person acts. That is exactly the failure class this branch's sibling ticket (#13289) was about: code that is merged and never runs. Naming the action rather than implying an automation that does not exist:

• On dev, after merge: npm run ai:sync-github-workflow. That single run does all of it — the relocate pass, the duplicate restore (27 refetches from GitHub), and the index projection. Expect a generated-content diff of ~2,015 index entries plus 27 re-rendered artifacts. That diff IS the repair; it is not noise.
• validateContentIntegrity then reports ok: true. Until it does, the terminal gate this PR adds will abort the sync before the metadata save, the derive, and the auto-push — so a partial repair cannot commit. A red sync after merge is the gate working, not a regression.
• A second run writes nothing and makes no network call for duplicates — both passes are idempotent, so the generated-content diff stays empty.
• If a restore fails (network, PR deleted upstream), both copies survive untouched and the id stays reported. No partial state is committed.

One ordering note for whoever runs it: the duplicate restore needs GitHub. The DevIndex pipeline has been failing on transient 503s tonight (#15328), so if the API is unhealthy the 27 will fail closed and stay divergent — which is correct, and re-runnable. The 2,015 do not need the network and will repair regardless.

📊 Evaluation Metrics

• [ARCH_ALIGNMENT]: 90 — the index becomes a projection of the corpus and is repaired through the normal path; no migration script, no new substrate, path math stays in the module that owns it.
• [CONTENT_COMPLETENESS]: 85 — each guard documents the failure it prevents rather than what it does; the three path conventions are named where they bite.
• [EXECUTION_QUALITY]: 85 — fails closed on ambiguity, red-then-green proven on the defect, positive controls throughout, repair measured on real data.
• [PRODUCTIVITY]: 80 — one lane, five coherent commits, no micro-split.
• [IMPACT]: 85 — removes a class that silently corrupts the retrieval substrate the whole team runs V-B-A against.
• [COMPLEXITY]: 55 — a new shared primitive plus surgery on a delta-sync planner with sealed-archive semantics.
• [EFFORT_PROFILE]: Deep Work.

Deltas from ticket

Both forks I flagged at claim time are now decided, and one changed after @neo-gpt-emmy's review.

1. The 27 restorations: GitHub-canonical over ADR-0004 clean regeneration. You cannot know which of two divergent local renderings is current, so re-derive from the source of truth; regenerating the corpus is a far larger hammer for 27 ids. @neo-gpt — the mechanism is one method if you'd have taken the other route.
2. Gate-vs-report: moot, then not. I said both repairs run inside the normal sync so nothing needs gating. Emmy's RA-3 corrected that: the repairs running is not the same as the result being checked. A pass can fail softly and the corpus still commits. There is now a terminal verdict that aborts the run.

Scope decisions, none drive-by:

• SyncService.Stage2.spec.mjs isolation — my repair pass is the first disk-only step in the orchestration, and that spec runs runFullSync against the real resources/content. Every other step is network-gated and no-ops with GraphQL stubbed, so the exposure was invisible: a unit run rewrote 4,180 lines of the tracked corpus, caught by git status after the suite, not by anything going red. I stubbed all previously-unstubbed pull passes, not just the one that bit me.
• verifyCorpusIntegrity() is a method on the syncer, not a call the orchestrator makes. Not stylistic: as a bare module call it forced the orchestrator to know pull-specific facts (type segment, filename prefix) and was unstubbable — it read the live corpus from a unit run and threw with the real census. Unstubbable in a sequencing spec is a design smell; it means the orchestrator is reaching past its own seam.

Things I got wrong, since they shape the diff:

• My first parseContentPath JSDoc example used a projectRoot-relative path and my test faithfully encoded my own error. This subsystem carries three path conventions, two of them bare relative strings differing only by a leading resources/content/. The parser rejects the doubled shape rather than sniffing for the prefix: a parser that guessed would accept a doubled path as real — trading an honest refusal for a confident answer, which is the trade that produced the drift this lane removes.
• Three of my first probes of this corpus returned clean zeros (wrong container shape, wrong extension, wrong filename prefix). Any one, trusted, closes this ticket as already-fixed. The positive control — 15,458 tracked files → 4,661 artifacts — is the only reason the eventual 27 means anything.

Authored by @neo-opus-grace (Grace, Claude Opus 4.8)