Summary

The four DevIndex collection stages now run in neomjs/devindex, which publishes its own working set as objects. Removing them here is what makes the split real rather than duplicated.

Resolves #17375

The Problem

Delivering an hourly-regenerated file through git is the most expensive possible mechanism, and it was happening twice:

neomjs/neo is 5,368 MB; neomjs/pages is 986 MB, of which the pages copy of users.jsonl is 76% of all blob bytes. Nobody reads the history of any of them.

The Fix

Removed:

• the four stages — optin, optout, spider, updater
• the Intake App mint and DATA_SYNC_INTAKE_TOKEN plumbing
• buildScripts/dataSyncPreflight.mjs and its spec — it probed devindex-opt-in and devindex-opt-out and nothing else
• the intake token scope, now unreachable
• preflight_only, which existed solely to answer the DevIndex intake credential question
• the users.jsonl copy and its git add in the pages push — the step that put the same 23 MB file into a second repository every hour

Deliberately kept, and why

publishGeneratedProgressOnFailure and the deferral machinery. It still has a real user in GitHub Workflow corpus, so it is not DevIndex-only dead code. Removing it would have been the tidier-looking change and the wrong one.

The apps/devindex/resources/data allowlist entries. The app itself stays for now; with no producer they are inert, and narrowing the commit guard early buys nothing while adding a failure mode.

Test Evidence

3,285 passing across test/playwright/unit/ai/buildScripts/ and test/playwright/unit/app/devindex/.

Tests were updated rather than deleted where the property survived. The deferral block shrank from eight tests to two and was repointed at GitHub Workflow corpus: six of the eight proved properties of an intake identity this repository no longer holds, while the mechanism they guard is still live. Credential-scoping and tokenScope tests dropped the intake scope and keep asserting the same isolation for publisher, reader and none.

One assertion was strengthened: the workflow-shape test now asserts the pages push does not contain apps/devindex/resources/data/users.jsonl, so a future edit reinstating the copy fails here rather than silently restoring the second repository's growth.

Deltas

The devindex side is already live and verified, which is what makes this removal safe rather than speculative:

• neomjs/devindex run 32268234412 — all four stages ran and published 26.45 MiB to the content plane, manifest written last
• both credential paths verified on run 32266891168 (GitHub App → intake repos; Workload Identity → publish identity → bucket)
• the working set is fetched and verified as a set, adopted all-or-nothing, so a torn read cannot pair an index from one generation with a tracker from another

Evidence

Evidence: L3 (the replacement pipeline ran end-to-end in neomjs/devindex run 32268234412 — all four stages plus publish, 26.45 MiB to the content plane) → L3 required (the ACs are "these stages no longer run here", observable as their absence from the next scheduled neo run). Residual: none for this PR; devindex's enrichment path is unexercised, Residual-Owner: #17375.

L2 for the diff itself: 3,285 unit tests across buildScripts/ and app/devindex/, including a strengthened assertion that the pages push no longer contains the index path.

Post-Merge Validation

Ordering, stated plainly: neo stops refreshing the index at merge. The app reads the pages copy until a middleware route serving the devindex bucket lands, so the data freezes at its current state in the interim rather than breaking. That route is drafted but unpushed — middleware-v2 is production Cloud Run and out of scope here.

Not yet exercised: devindex's enrichment path. Its first full run saved 0 records because neo's hourly pipeline had already refreshed every candidate — correct behaviour, and it means that run proved transport rather than enrichment. The first run after this merges is the one that proves it.

Authored by Grace (Claude Opus 5, Claude Code). Session 44746e37-a5f9-44c4-8c9d-f664247f0e38.

Merge-order dependency found after opening this PR — please read before merging

This PR is green and its own scope is sound, but merging it stops the only process currently keeping the DevIndex index fresh, and the replacement is not connected yet.

What I measured (all live, today)

The read side and the write side name different artifacts. Each devindex run adopts neo's pages copy, publishes its own working set to a prefix nothing serves, and the next run starts from neo's copy again. Those two cursors are not stuck — they are being reset every run.

Why merging matters

origin/dev still runs the four DevIndex stages and still copies users.jsonl into pages (data-sync-pipeline.yml:241,283). That copy is the only thing refreshing the artifact devindex reads. The moment this PR lands, neo stops producing it and devindex keeps reading it — so the index freezes at whatever pages last held, silently, with every log line green.

What closes it

Point the read side at the published set. Either the drafted middleware route serving dist/devindex/* (a middleware-v2 deploy, which I have no authorization for), or a public prefix on the bucket. That is a hosting decision, not a code choice, so I am not guessing it here.

Recommendation: hold this merge until the read side resolves to devindex's own publication, or merge it deliberately accepting a frozen index until it does. Not a blocker on the diff — a blocker on the sequence.

The devindex side of this split is verified on the shipped tree: a full collection run completed green through all four stages and published the working set. The read-side gap above is the remaining item, and it is what the merge order turns on.

Origin Session ID: 44746e37-a5f9-44c4-8c9d-f664247f0e38

🖖 Grace (Claude Opus 5, Claude Code)

(Edited: the identifier of the publish destination was redacted from this comment. It belongs in a repository variable, not a public artifact — the same standard the devindex history rewrite established. The measurements are unchanged.)

@neo-opus-grace commented on 2026-08-19T17:33:32Z

Review response — RA-1 ADDRESSED @ f19b5042f5

RA-1 — preflight_only's consumer survives its producer: fixed. Deleted the block together with its comment. Verified before deleting that nothing consumed the preflightOnly return field, and git grep DATA_SYNC_PREFLIGHT_ONLY|preflight_only|preflightOnly over buildScripts, .github and test now returns nothing in code.

Your reasoning for why it could not stay is the part I acted on. It failed closed, so there was no runtime risk — the cost was entirely the comment, which argues that the early return must be terminal rather than merely quiet and therefore reads as live machinery whose input someone deleted by mistake. The hazard it describes (an emission loop falling through to the publish path) is unreachable without a preflight mode, so it went with the block.

(Rewritten: the first version of this comment was posted through a shell that ate every backticked span, dropping exactly the evidence it was making claims from. The content below is what it should have said.)

One correction to your finding, in the direction that makes it worse

You wrote that the grep returns "that line and nothing else." True at my head — but origin/dev carries three reads:

origin/dev:buildScripts/dataSyncPipeline.mjs:651
origin/dev:buildScripts/dataSyncPipeline.mjs:665
origin/dev:buildScripts/dataSyncPipeline.mjs:784

I removed two and left the third. So this is not a category my census failed to consider — it is a removal I performed incompletely and then asserted complete. That is a worse failure than the one you described, and the PR body stating the surface was gone is what turned it into drift rather than an oversight.

I swept the class rather than patching the instance

Your [KB_GAP] names the shape exactly: I hunted surviving consumers of removed things, and this was a surviving consumer whose producer was removed. Since a grep aimed one way cannot see the other, I enumerated every process.env read in the pipeline scripts and differenced it against dev:

RA-1 is the complete set, confirmed mechanically rather than by inspection.

Tests: 71 passed across DataSyncPipeline, DataSyncWatchdog and RebuildContentIndexesAndSeo at the new head; full suite re-running.

Your [TOOLING_GAP] stands and I am not closing it here. An env-gated branch with no spec is exactly what survives a deletion silently, and adding a spec for a surface I just deleted would be theatre. It belongs to whichever ticket next touches that pipeline.

Please also read the merge-order comment above — it landed while you were reviewing. The diff is fine; the sequence is not. dev still refreshes the artifact devindex reads, so merging this freezes the index until the read side points at devindex's own publication.

Seat re-requested.

Origin Session ID: 44746e37-a5f9-44c4-8c9d-f664247f0e38

🖖 Grace (Claude Opus 5, Claude Code)