Sub 2 of #17238, and the ticket where the growth actually stops. neomjs/devindex exists, the application and its guides and unit suite already live there, and neo's copy is still the one the pipeline drives.
#17238 measured users.jsonl at 90.5% of neo's blob bytes, and all devindex paths at 95.6%. Re-measured 2026-08-19: neo 5.21 GiB, neomjs/pages0.96 GiB, 1,790 lifetime commits to that single file.
These two changes are one ticket on purpose.#17238's own body names "relocating DevIndex without changing the storage model" as the most attractive wrong answer, with neomjs/pages as the live proof of that failure mode — a fresh repository committing this file hourly reaches multiple GiB inside six months. Building the artifact-publish in neo and then moving it a week later is that mistake with an extra step. The new repository has no pipeline to retrofit, which makes greenfield strictly cheaper than sequencing them apart.
The Problem
Two separable problems that must be solved together.
1. The collection stages do not belong here.buildScripts/dataSyncPipeline.mjs runs seven emission stages. Four are DevIndex:
stage
tokenScope
install dependencies
none
GitHub Workflow corpus
reader
DevIndex Opt-In
intake
DevIndex Opt-Out
intake
DevIndex Spider
intake
DevIndex Updater
intake
content indexes and SEO
reader
Every DevIndex stage uses intake; no neo stage does. The Intake token is minted for devindex-opt-in and devindex-opt-out (data-sync-pipeline.yml). The credential axis is a clean cut, not an untangle.
2. Whatever runs them must not commit what they generate. Otherwise the disease relocates with the app.
The Architectural Reality
What moves. Four stages, and exactly one of six copy operations in Push Data to neomjs/pages — cp apps/devindex/resources/data/users.jsonl …. The other five (portal data, workstation.png, sitemap.xml + llms.txt, learn/**, resources/content/** across six buckets) are neo-corpus and stay. This is a pipeline split, not a move.
What retires rather than moves. Every DevIndex stage carries publishGeneratedProgressOnFailure: true plus the deferredError rethrow. Its own comment records why: a denial on the first DevIndex stage threw out of the loop, so content indexes and SEO never ran and the corpus was discarded unpublished — freezing resources/content/** on dev for nineteen hours. That machinery exists only because DevIndex enrichment shares a process with neo's corpus publication. Split the process and a DevIndex failure structurally cannot reach neo's corpus. The flag has nothing left to protect and should not be ported.
What must not enter git, and what must. Measured on disk:
users.jsonl · tracker.json · visited.json 26.5 MiB DERIVED — the Spider rebuilds it
allowlist · blocklist · optin-sync
optout-sync · threshold · failed 543 B CURATED — irreplaceable
A 51,000× ratio, and only the small half is information. Losing the derived half costs Spider time; losing blocklist.json loses somebody's opt-out decision. The curated 543 bytes stay in git — for this project the opt-out audit trail is an ethical asset, not merely data. The derived 26.5 MiB never enters git again, in any repository.
Why publishing without a commit is possible at all.#17374 establishes the read side: the pipeline obtains its previous state from the published artifact over HTTPS, verified live (200, etag, accept-ranges, open CORS). Once the producer no longer reads from git, nothing requires it to write to git either.
Hosting is a late binding. The shape is identical whether the destination is GitHub Pages' artifact deploy, an upload to whatever serves neomjs.com (x-powered-by: Express — it is not bare Pages), or object storage. Every candidate supports publish-without-commit; only the branch-based Pages source does not, and that source is precisely the mechanism that built the 985 MB. Hosting decides the workflow's final step, not the design.
The Fix
a devindex-owned workflow runs the four intake-scoped stages, on the same hourly cadence
it publishes the derived artifacts without a commit, to the destination the operator selects
.gitignore in neomjs/devindex excludes the derived three; the curated six stay tracked
neo's dataSyncPipeline.mjs drops the four stages, the Intake token mint, and the users.jsonl copy
publishGeneratedProgressOnFailure / deferredError are removed from the moved stages rather than ported
neo deletes learn/guides/devindex/** and apps/devindex/**
the checkout drops to fetch-depth: 1 once nothing here pushes generated data (see below)
Contract Ledger Matrix
Target Surface
Source of Authority
Proposed Behavior
Fallback
Docs
Evidence
DevIndex emission stages
buildScripts/dataSyncPipeline.mjs
the four intake stages are removed from neo and defined in the devindex workflow
none — a stage that runs in neither repository is a gap, not a degradation
the Data Factory guide's Orchestrator page
a fixture asserting neo's stage list contains no intake-scoped entry
Intake installation token
data-sync-pipeline.yml token mint
minted by the devindex workflow; neo mints reader + publisher only
none
the workflow's own comments
neo's mint step no longer requests devindex-opt-in / devindex-opt-out
derived artifacts in git
.gitignore in neomjs/devindex
users.jsonl, tracker.json, visited.json are never committed
none by design — a committed copy is the defect
the repository README
a guard failing if any derived path is tracked
curated inputs in git
same
the six small files stay tracked and reviewable
none
as above
the same guard asserting they ARE tracked, so the rule cannot be satisfied by ignoring everything
Decision Record impact
none. Structure-map gate executed; N/A for Agent OS placement — surfaces are apps/devindex/**, buildScripts/dataSyncPipeline.mjs, .github/workflows/data-sync-pipeline.yml and the new devindex-side workflow.
Acceptance Criteria
AC-1: the four intake-scoped stages run from a devindex-owned workflow on the hourly cadence, and neo's stage list contains no intake-scoped entry. A spec asserts the absence rather than a reviewer reading the list.
AC-0 (added 2026-08-19; DONE — and its second half was never the blocker I claimed): no consumer in neomjs/devindex depends on the checkout copy being committed. Producer: neomjs/devindex@72a7043. Delivery: neomjs/devindex@a7dcd27 — npm install fetches the published index (~10 MB gzipped, ~1.4 s) to the path the app already reads. store/Contributors.mjs was never changed and never needed to be — see the corrected note below.
AC-2 (DONE):neomjs/devindex@a7dcd27 — users.jsonl, tracker.json and visited.json are gitignored and git rm --cacheded; buildScripts/checkDataTracking.mjs fails if any is tracked. Verified failing on a re-added derived file, then green.
AC-3 (DONE): the same guard asserts the curated seven ARE tracked, and was verified failing on a removed blocklist.json — the shortcut it exists to block. It immediately earned itself by catching that index-provenance.json did not yet exist; that was seeded through Storage.recordIndexProvenance so the format cannot drift from what the producer writes.
AC-4:publishGeneratedProgressOnFailure and the deferredError arm are absent from the moved stages, and the reason is recorded where a future reader will meet it. Porting them is the defect: after the split there is no shared corpus publication for a DevIndex failure to abort.
AC-5: neo's Push Data to neomjs/pages no longer copies users.jsonl, and the other five copy operations are untouched. A spec pins the five, so a later edit cannot quietly widen this removal.
AC-6:learn/guides/devindex/** and apps/devindex/** are removed from neo, and no neo surface references them — swept with a stated file set and a control proving the sweep can reach a planted reference.
AC-6b:data-sync-pipeline.yml checks out at fetch-depth: 1. Raised in review of #17378: actions/checkout defaults to shallow and this workflow explicitly sets fetch-depth: 0 — the only option on that step with no justifying comment. The git work the pipeline does (rev-parse, reset --hard origin/dev, diff --cached, SHA equality rather than ancestry) needs no history; the depth is presumed to exist for the publish push, since remotes reject shallow updates. This ticket removes that push, so the presumption is testable here and nowhere else. If shallowing still fails, record why — an uncommented option that turns out to be load-bearing deserves the comment it never had. Expected payoff: the 239 s Checkout repository step of a 22.6 min run.
AC-7 (post-merge): one full hourly cycle completes end to end from the devindex workflow, and the served artifact's last-modified advances. Flagged post-merge: no pre-merge evidence can observe a scheduled run.
Correction, 2026-08-19 — this ticket counted one consumer and there are two
The Architectural Reality above says "The browser has always read this file over HTTPS from the deployed site; only the producer read it from git." That is true of neomjs/neo, where the browser fetches the deployed copy, and it is false of neomjs/devindex as it stands. Measured:
basePath-relative resolves against whatever is serving the app — which, in local development, is the checkout. So in this repository the checkout has two readers, not one, and AC-2's guard would have gone green while the grid rendered empty for the operator on localhost:8082 and for anyone cloning fresh.
I inherited the single-consumer framing from #17374, where it was correct, and did not re-check it against the repository this ticket actually targets.
Second correction, same day — my remedy was wrong even though the observation was right
I wrote that the browser consumer was "the remaining blocker" and gated AC-2 on it. It was not a blocker, and the operator's reframing is what exposed why:local and committed are separable axes. The bloat never came from the file being on disk — it came from committing it hourly. So the store's basePath-relative URL was correct all along; the only thing missing was something to put the file at that path once git stopped.
npm run devindex:pull-data (also run from postinstall) does that in ~1.4 s, and store/Contributors.mjs is unchanged. Fetching remotely at runtime — which is where my "blocker" framing was heading — would have moved the cost from one download per developer to one per visitor per page load, which is worse on every axis including the one that motivated the ticket.
The general shape worth keeping: an observation can be correct and its implied remedy still wrong. I measured a real dependency and then reached for the expensive way to break it.
What the same measurement pass also found
The file is committed hourly into two repositories, not one. In neomjs/pages it lands at node_modules/neo.mjs/apps/devindex/resources/data/users.jsonl — which is exactly the URL the app fetches, proxied by the Cloud Run middleware. A blob census there (against a 46-day-stale clone, so a floor) attributes 75.8% of all blob bytes to that one path. GitHub reports neomjs/pages at 986 MB and neomjs/neo at 5,368 MB; neomjs/devindex is 11 MB and now stays there.
This widens AC-5: neo's copy step is not the only writer that has to stop.
This does not change the design, and it is worth saying why rather than leaving it implied: the browser fix is the same late binding as the producer's — point the store at the published artifact instead of a checkout-relative path. It is blocked on the same hosting decision and lands with it, not before it.
Out of Scope
Reclaiming existing history. Axis 3. This stops accrual; it reclaims nothing, and neo stays at 5.21 GiB until that lands.
neomjs/pages growth from the neo corpus. Removing DevIndex stops DevIndex's contribution; resources/content/** and portal data keep the curve going. Same mechanism would fix it and that is the larger prize, but it is neo's corpus, not this app's.
Relocating without the storage model. The named most-attractive-wrong-answer of #17238. neomjs/pages at 0.96 GiB is the proof, and it is five months behind neo on the identical curve.
Building the artifact-publish in neo first, then moving it. Correct in sequence, wasteful in fact: the new repository has no pipeline to retrofit.
Porting the deferral machinery because it looks like hardening. It is a workaround for a coupling this ticket deletes. Carrying it forward preserves a scar with no wound.
.gitignore-ing the whole data directory. It satisfies AC-2 and silently discards the opt-out audit trail — the one artifact in there that cannot be regenerated. AC-3 exists to make that failure loud.
Waiting on the hosting decision to start. The design is destination-independent; only the final publish step differs.
Related
Parent: #17238 · depends on #17374 (the read side) · Axis 3 depends on this.
Retrieval Hint: query_raw_memories("devindex pipeline split intake token four stages deferredError retire") · "publish without commit derived versus curated"
Live latest-open sweep: checked the latest 20 open issues at 2026-08-19T08:36:42Z plus a 30-message A2A claim scan across all read-states; no equivalent ticket and no in-flight claim on this scope.
Context
Sub 2 of #17238, and the ticket where the growth actually stops.
neomjs/devindexexists, the application and its guides and unit suite already live there, and neo's copy is still the one the pipeline drives.#17238measuredusers.jsonlat 90.5% of neo's blob bytes, and all devindex paths at 95.6%. Re-measured 2026-08-19: neo 5.21 GiB,neomjs/pages0.96 GiB, 1,790 lifetime commits to that single file.These two changes are one ticket on purpose. #17238's own body names "relocating DevIndex without changing the storage model" as the most attractive wrong answer, with
neomjs/pagesas the live proof of that failure mode — a fresh repository committing this file hourly reaches multiple GiB inside six months. Building the artifact-publish in neo and then moving it a week later is that mistake with an extra step. The new repository has no pipeline to retrofit, which makes greenfield strictly cheaper than sequencing them apart.The Problem
Two separable problems that must be solved together.
1. The collection stages do not belong here.
buildScripts/dataSyncPipeline.mjsruns seven emission stages. Four are DevIndex:nonereaderintakeintakeintakeintakereaderEvery DevIndex stage uses
intake; no neo stage does. The Intake token is minted fordevindex-opt-inanddevindex-opt-out(data-sync-pipeline.yml). The credential axis is a clean cut, not an untangle.2. Whatever runs them must not commit what they generate. Otherwise the disease relocates with the app.
The Architectural Reality
What moves. Four stages, and exactly one of six copy operations in
Push Data to neomjs/pages—cp apps/devindex/resources/data/users.jsonl …. The other five (portal data,workstation.png,sitemap.xml+llms.txt,learn/**,resources/content/**across six buckets) are neo-corpus and stay. This is a pipeline split, not a move.What retires rather than moves. Every DevIndex stage carries
publishGeneratedProgressOnFailure: trueplus thedeferredErrorrethrow. Its own comment records why: a denial on the first DevIndex stage threw out of the loop, socontent indexes and SEOnever ran and the corpus was discarded unpublished — freezingresources/content/**ondevfor nineteen hours. That machinery exists only because DevIndex enrichment shares a process with neo's corpus publication. Split the process and a DevIndex failure structurally cannot reach neo's corpus. The flag has nothing left to protect and should not be ported.What must not enter git, and what must. Measured on disk:
A 51,000× ratio, and only the small half is information. Losing the derived half costs Spider time; losing
blocklist.jsonloses somebody's opt-out decision. The curated 543 bytes stay in git — for this project the opt-out audit trail is an ethical asset, not merely data. The derived 26.5 MiB never enters git again, in any repository.Why publishing without a commit is possible at all.
#17374establishes the read side: the pipeline obtains its previous state from the published artifact over HTTPS, verified live (200,etag,accept-ranges, open CORS). Once the producer no longer reads from git, nothing requires it to write to git either.Hosting is a late binding. The shape is identical whether the destination is GitHub Pages' artifact deploy, an upload to whatever serves
neomjs.com(x-powered-by: Express— it is not bare Pages), or object storage. Every candidate supports publish-without-commit; only the branch-based Pages source does not, and that source is precisely the mechanism that built the 985 MB. Hosting decides the workflow's final step, not the design.The Fix
intake-scoped stages, on the same hourly cadence.gitignoreinneomjs/devindexexcludes the derived three; the curated six stay trackeddataSyncPipeline.mjsdrops the four stages, the Intake token mint, and theusers.jsonlcopypublishGeneratedProgressOnFailure/deferredErrorare removed from the moved stages rather than portedlearn/guides/devindex/**andapps/devindex/**fetch-depth: 1once nothing here pushes generated data (see below)Contract Ledger Matrix
buildScripts/dataSyncPipeline.mjsintakestages are removed from neo and defined in the devindex workflowintake-scoped entrydata-sync-pipeline.ymltoken mintdevindex-opt-in/devindex-opt-out.gitignoreinneomjs/devindexusers.jsonl,tracker.json,visited.jsonare never committedDecision Record impact
none. Structure-map gate executed; N/A for Agent OS placement — surfaces areapps/devindex/**,buildScripts/dataSyncPipeline.mjs,.github/workflows/data-sync-pipeline.ymland the new devindex-side workflow.Acceptance Criteria
intake-scoped stages run from a devindex-owned workflow on the hourly cadence, and neo's stage list contains nointake-scoped entry. A spec asserts the absence rather than a reviewer reading the list.neomjs/devindexdepends on the checkout copy being committed. Producer:neomjs/devindex@72a7043. Delivery:neomjs/devindex@a7dcd27—npm installfetches the published index (~10 MB gzipped, ~1.4 s) to the path the app already reads.store/Contributors.mjswas never changed and never needed to be — see the corrected note below.neomjs/devindex@a7dcd27—users.jsonl,tracker.jsonandvisited.jsonare gitignored andgit rm --cacheded;buildScripts/checkDataTracking.mjsfails if any is tracked. Verified failing on a re-added derived file, then green.blocklist.json— the shortcut it exists to block. It immediately earned itself by catching thatindex-provenance.jsondid not yet exist; that was seeded throughStorage.recordIndexProvenanceso the format cannot drift from what the producer writes.publishGeneratedProgressOnFailureand thedeferredErrorarm are absent from the moved stages, and the reason is recorded where a future reader will meet it. Porting them is the defect: after the split there is no shared corpus publication for a DevIndex failure to abort.Push Data to neomjs/pagesno longer copiesusers.jsonl, and the other five copy operations are untouched. A spec pins the five, so a later edit cannot quietly widen this removal.learn/guides/devindex/**andapps/devindex/**are removed from neo, and no neo surface references them — swept with a stated file set and a control proving the sweep can reach a planted reference.data-sync-pipeline.ymlchecks out atfetch-depth: 1. Raised in review of #17378:actions/checkoutdefaults to shallow and this workflow explicitly setsfetch-depth: 0— the only option on that step with no justifying comment. The git work the pipeline does (rev-parse,reset --hard origin/dev,diff --cached, SHA equality rather than ancestry) needs no history; the depth is presumed to exist for the publish push, since remotes reject shallow updates. This ticket removes that push, so the presumption is testable here and nowhere else. If shallowing still fails, record why — an uncommented option that turns out to be load-bearing deserves the comment it never had. Expected payoff: the 239 sCheckout repositorystep of a 22.6 min run.last-modifiedadvances. Flagged post-merge: no pre-merge evidence can observe a scheduled run.Correction, 2026-08-19 — this ticket counted one consumer and there are two
The Architectural Reality above says "The browser has always read this file over HTTPS from the deployed site; only the producer read it from git." That is true of
neomjs/neo, where the browser fetches the deployed copy, and it is false ofneomjs/devindexas it stands. Measured:apps/devindex/store/Contributors.mjs:50 url: Neo.config.basePath + 'apps/devindex/resources/data/users.jsonl'basePath-relative resolves against whatever is serving the app — which, in local development, is the checkout. So in this repository the checkout has two readers, not one, and AC-2's guard would have gone green while the grid rendered empty for the operator onlocalhost:8082and for anyone cloning fresh.I inherited the single-consumer framing from #17374, where it was correct, and did not re-check it against the repository this ticket actually targets.
Second correction, same day — my remedy was wrong even though the observation was right
I wrote that the browser consumer was "the remaining blocker" and gated AC-2 on it. It was not a blocker, and the operator's reframing is what exposed why: local and committed are separable axes. The bloat never came from the file being on disk — it came from committing it hourly. So the store's
basePath-relative URL was correct all along; the only thing missing was something to put the file at that path once git stopped.npm run devindex:pull-data(also run frompostinstall) does that in ~1.4 s, andstore/Contributors.mjsis unchanged. Fetching remotely at runtime — which is where my "blocker" framing was heading — would have moved the cost from one download per developer to one per visitor per page load, which is worse on every axis including the one that motivated the ticket.The general shape worth keeping: an observation can be correct and its implied remedy still wrong. I measured a real dependency and then reached for the expensive way to break it.
What the same measurement pass also found
The file is committed hourly into two repositories, not one. In
neomjs/pagesit lands atnode_modules/neo.mjs/apps/devindex/resources/data/users.jsonl— which is exactly the URL the app fetches, proxied by the Cloud Run middleware. A blob census there (against a 46-day-stale clone, so a floor) attributes 75.8% of all blob bytes to that one path. GitHub reportsneomjs/pagesat 986 MB andneomjs/neoat 5,368 MB;neomjs/devindexis 11 MB and now stays there.This widens AC-5: neo's copy step is not the only writer that has to stop.
This does not change the design, and it is worth saying why rather than leaving it implied: the browser fix is the same late binding as the producer's — point the store at the published artifact instead of a checkout-relative path. It is blocked on the same hosting decision and lands with it, not before it.
Out of Scope
neomjs/pagesgrowth from the neo corpus. Removing DevIndex stops DevIndex's contribution;resources/content/**and portal data keep the curve going. Same mechanism would fix it and that is the larger prize, but it is neo's corpus, not this app's.Avoided Traps
neomjs/pagesat 0.96 GiB is the proof, and it is five months behind neo on the identical curve..gitignore-ing the whole data directory. It satisfies AC-2 and silently discards the opt-out audit trail — the one artifact in there that cannot be regenerated. AC-3 exists to make that failure loud.Related
Parent: #17238 · depends on #17374 (the read side) · Axis 3 depends on this.
Retrieval Hint:
query_raw_memories("devindex pipeline split intake token four stages deferredError retire")·"publish without commit derived versus curated"Live latest-open sweep: checked the latest 20 open issues at 2026-08-19T08:36:42Z plus a 30-message A2A claim scan across all read-states; no equivalent ticket and no in-flight claim on this scope.
Origin Session ID: a105d215-c261-4b34-82a9-546596f665ef