The Data Sync Pipeline uses git history as a state database — 95.6% of neo's 3.8 GiB pack is one hourly-rewritten file

Problem scope

apps/devindex/resources/data/users.jsonl is a 23 MiB / 49,999-line generated artifact that the Data Sync Pipeline regenerates and commits to dev on an hourly schedule. Git history is append-only, so every regeneration is permanent. The pipeline only ever reads the latest state — Storage.readJson(config.paths.users) reads the working tree that actions/checkout populates — and git faithfully retains all 1,767 of them.

Census — measured locally over --all refs, 2026-08-16, local HEAD a358a6668a, origin/dev @ bd4ec27536:

Reproduce: git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize:disk) %(rest)', aggregate objectsize:disk by path.

This is not a git-tuning problem — corrected 2026-08-16

A first version of this section claimed delta compression was 'working essentially perfectly'. That was measured against the wrong baseline and is withdrawn. The raw-vs-disk ratio (39.2 GiB raw stored in 3.3 GiB, 1,761 of 1,768 versions stored as deltas) compares against no compression at all, which nothing was ever going to cost. The right baseline is what a correct delta chain would produce for this file.

Re-measured, same tree:

The waste has a identifiable cause: delta-base selection has a heavy tail. Median base distance is 2 revisions, but the mean is 190.7 and p90 is 772 — git orders delta candidates by type and size, and 1,768 blobs all within a few hundred bytes of 24 MiB sort effectively arbitrarily among themselves, so a substantial minority get paired against revisions hundreds of commits away, where thousands of lines genuinely differ.

The conclusion is unchanged, and now rests on the right fact. Even if delta selection were perfect, ~525 KiB per sync × ~10.7 syncs/day is ~165 MiB/month, permanently. The file is a ranked leaderboard: ranks shift, so ~1.55% of its lines really do change every hour. No packing strategy fixes a file whose content actually changes, which is why an aggressive repack (--window=250 --depth=250 --window-memory=2g) moves the number by nothing.

Two consequences follow that did not follow from the withdrawn version:

1. Dropping the file's history while continuing to commit it buys a one-time reclaim only. Regrowth resumes immediately at ~165 MiB/month floor, or ~0.55 GiB/month at the observed delta quality.
2. The absolute figure is pack-state dependent — it moves with repacking — so any headline size number is a property of a particular pack, not of the history. It never gets small, which is the point.

The fix has to be structural: the artifact must stop entering git history at all.

Three costs beyond clone size

1. npm. users.jsonl (22.97 MiB) is the single largest file in the published tarball; devindex data totals 26.5 of 96.2 MiB unpacked. .npmignore:4 excludes apps/devindex/resources/*.json — the former path — and does not cover the current resources/data/*.jsonl. Every npm i neo.mjs downloads contributor data.
2. CI, hourly and compounding. Of a 22.6 min average run, Checkout repository is 239 s and Push Data to neomjs/pages is 128 s — 27% of the run, ~2.4 h/day, growing monotonically. The pipeline pays for its own bloat every hour (run 31955681881).
3. neomjs/pages has the identical disease. data-sync-pipeline.yml:283 git adds the same file into that repo hourly. pages is at 0.95 GiB — the same trajectory, roughly five months behind. Any fix that treats only neomjs/neo leaves the second instance running.

The cost was already paid downstream — twice, undiagnosed

• #16546 (closed): the Memory Core tenant mirror cloned this repo at 4.9 GB and OOM-killed the orchestrator every ~290 s. Resolved with a blobless clone.
• #16557 (open): that blobless workaround turns first ingestion into 23,931 network round trips.

Both tickets treat the consequence; neither names the cause. A Memory Core sweep surfaced no prior size analysis of the corpus at all. That two subsystems absorbed the blast radius without the root cause ever being filed is the friction→gold signal here, and it is the reason this is an Epic rather than a cleanup commit.

Why an Epic, not a single ticket

The work splits into three axes that differ in urgency, reversibility and authority, and cannot be delivered by one PR:

• Stopping the accrual is urgent, cheap and reversible.
• Relocating DevIndex is an architectural move with an operator-set destination, and requires splitting a pipeline that is not DevIndex-only.
• Reclaiming the existing 3.4 GiB is irreversible, public-facing, and touches every fork and every SHA citation in our substrate.

Sequencing is load-bearing: the purge scope for the third axis depends on what the first two leave behind, so it must land last.

Intended solution shape

1 — The published artifact becomes the single source of truth. Today users.jsonl exists three times: neo's git, pages' git, and the served copy. Only the served copy is ever read by the application. The convergent shape is that the pipeline fetches the previous state from the published artifact, mutates it, and republishes it — so git, in any repo, holds only code. This also removes the drift risk between the two committed copies. Integrity-on-read and an artifact-loss recovery path are part of this shape, not afterthoughts.

2 — DevIndex relocates to a standalone neomjs/devindex repository. Operator-set goal, recorded here for the first time: DevIndex is a product with its own intake repositories (devindex-opt-in, devindex-opt-out), its own data lifecycle and its own release cadence, and it currently taxes the engine's npm tarball. The relocation must carry the new storage model with it — a fresh repository committing this file hourly reaches 3.4 GB inside six months, which is precisely what neomjs/pages demonstrates today.

A complication that shapes the decomposition: data-sync-pipeline.yml is not DevIndex-only. The same workflow emits resources/content/** (the issue/PR/discussion corpus), portal data, sitemap.xml, llms.txt and learn/**. This is a pipeline split — DevIndex collection and publication leave, neo-corpus emission and SEO stay — not a pipeline move.

3 — History rewrite to reclaim the existing pack. Purging the generated-data paths (apps/devindex/resources/**, apps/devrank/resources/**, resources/content/.sync-metadata.json at 28.3 MiB, and the generated portal artifacts at ~14 MiB) takes .git from 3.8 GiB to ~170 MiB. git filter-repo --invert-paths, then a force-push of all refs.

The blast radius this axis must carry, rather than discover:

• Every SHA from 2026-02-12 forward changes. Our substrate cites exact heads pervasively — review bodies, tickets, ADRs, Memory Core entries. filter-repo emits a commit-map; preserving that map as permanent public substrate is the workable answer, not rewriting citations.
• 231 forks, 3,253 stars, and every open PR invalidated. This lands at a release boundary with the open-PR queue drained, or it does not land.
• A force-push does not shrink GitHub's diskUsage. Unreachable objects survive in the fork network until GitHub Support runs gc. Fresh clones go small immediately — that is the real win — but the public size figure lags, and communication must say so.
• Downstream re-clones: the Memory Core tenant mirror (#16546 / #16557) and KB ingestion.

Because this axis is irreversible and affects every family's substrate citations, it requires an Ideation Sandbox consensus round before execution. An Epic body is not sufficient authority for an irreversible rewrite of a public repository, and execution remains with the human operator per §critical_gates.

Out of scope

• The DevIndex product roadmap — collection heuristics, threshold tuning, opt-in/opt-out flows. This Epic touches those services only where storage or repository placement forces it.
• neomjs/pages as a deployment mechanism. Its git growth is in scope; replacing GitHub Pages is not.
• The blobless-clone round-trip defect (#16557). It stays independently valid: a mirror should not pay for history it never reads, whatever this repo's size becomes.
• Retroactive correction of SHA citations across existing substrate.

Avoided traps / rejected shapes

• Git-level tuning (pack.depth, bigFileThreshold, aggressive gc). Falsified above: compression is already at 91.6%; there is no setting left to turn.
• Relocating DevIndex without changing the storage model. This is the most attractive wrong answer, because it looks like a complete fix while re-creating the defect in a fresh repository. neomjs/pages is the live proof of that failure mode.
• Rewriting history first. It is the visible number, so it draws attention first — but purge scope depends on what the other two axes leave behind, and rewriting a still-growing repository buys a number that immediately starts degrading again.
• .npmignore-only. Fixes the tarball, leaves clone size, CI cost and the pages copy untouched.
• Treating neomjs/neo alone. neomjs/pages carries the same hourly commit and is five months behind on the same curve.

Substrate notes

Structure-map gate executed (npm run --silent ai:structure-map -- --files --loc); recorded N/A for Agent OS placement — this Epic places no new ai/ substrate. Its surfaces are apps/devindex/**, buildScripts/dataSyncPipeline.mjs, .github/workflows/data-sync-pipeline.yml and .npmignore.

Not Discussion-graduated: filed on direct operator direction ("5GB is not acceptable"; standalone neomjs/devindex as the set destination). The third axis carries its own graduation requirement, stated above.

Sub-tickets are attached via the parent-child relationship graph and added incrementally; this body deliberately carries no sub registry.