The Data Factory reads its previous index from git, never from the web

Context

Sub 1 of #17238, and the only one of its three axes that is blocked by nothing.

#17238 measured that apps/devindex/resources/data/users.jsonl is 90.5% of all blob bytes in this repository. Re-measured 2026-08-19, three days after that census: GitHub-side size 5.12 → 5.21 GiB, local .git 3.8 → 4.1 GiB, 25 further hourly commits to that one file (1,790 lifetime). The census aged ~90 MB while it was being discussed.

This ticket does not stop that. It changes where the pipeline reads from, which is the safe half of the storage-model change and the prerequisite for the half that does.

The Problem

Storage reads the previous index out of the working tree that actions/checkout populates. That is why the pipeline needs a checkout of a 4 GiB repository to obtain one 24 MB file it only ever reads the tip of — Checkout repository is 239 s of a 22.6 min run per #17238's measurement.

The read is also the thing that couples the pipeline to git at all. As long as the previous state is obtained from the repository, publishing without committing is impossible: the next run would have nothing to mutate.

The asymmetry worth naming: the browser has always read this file over HTTPS from the deployed site. Only the producer reads it from git. The consumer's path is already the one this ticket adopts.

The Architectural Reality

Verified live against the current deployment on 2026-08-19:

GET https://neomjs.com/node_modules/neo.mjs/apps/devindex/resources/data/users.jsonl
HTTP/2 200 · content-length: 24084184
etag: "6a856439-16f7ed8"        strong validator
accept-ranges: bytes            resumable / partial reads
access-control-allow-origin: *  readable cross-origin
last-modified: Wed, 19 Aug 2026 08:07:21 GMT

Every primitive this needs already exists on the serving side. Nothing has to be built to host the read; only the reader changes.

The change belongs in apps/devindex/services/Storage.mjs — the flat-file abstraction the CLI stages already go through (Storage.readJson(config.paths.users)) — not in buildScripts/dataSyncPipeline.mjs or the workflow. That placement is load-bearing for sequencing: Storage.mjs travels with the app when #17238's Axis 2 relocates it, so this work is not thrown away by the move. A workflow-level implementation would be.

The Fix

Storage obtains the previous index by fetching the published artifact, and mutates that. Writing, and committing, stay exactly as they are today.

• fetch the published URL; on success, that is the previous state
• record the served etag alongside the write, so the next run can assert it is reading back what this run published
• on fetch failure, fall back to the checkout copy and say so loudly — a silent fallback would hide the very coupling this removes
• the small curated files (allowlist, blocklist, optin-sync, optout-sync, threshold, failed — 543 bytes in total) are untouched by this ticket and keep coming from git, where they belong

Contract Ledger Matrix

Decision Record impact

none — this changes where an existing service reads from. It introduces no ai/ substrate and no new architectural primitive. Structure-map gate executed (npm run --silent ai:structure-map -- --files --loc); recorded N/A for Agent OS placement, as the surfaces are apps/devindex/services/**.

Acceptance Criteria

• AC-1: Storage returns the previous index from the published artifact, and a fixture proves the fetched path and the checkout path yield byte-identical output for identical input — so the current behaviour is this change's own control.
• AC-2: a fetch failure falls back to the checkout copy and emits an explicit failure line naming the URL and the cause. A spec asserts the fallback is never silent; a silent fallback would let the git coupling persist invisibly after this ticket claims to have removed it.
• AC-3 (amended during implementation — original struck below): a content digest over the exact bytes published is recorded with each publish, and a subsequent run whose fetched digest does not match the recorded one falls back to the checkout copy rather than mutating an artifact it did not publish. A spec asserts the fallback, and separately asserts that a first run with no recorded digest proceeds — absence is not mismatch.

  • AC-3 (original): the served etag is recorded with each publish, and a subsequent run whose fetched etag does not match the recorded one refuses rather than mutating an artifact it did not publish. A spec asserts refusal, and separately asserts that a first run with no recorded etag proceeds — absence is not mismatch.

  • Why it moved, on two axes. Refuses → falls back: a propagation lag presents exactly as a mismatch — a stale CDN copy returns 200 with a real body and a wrong digest — so refusal would wedge the pipeline on the ordinary case. The checkout copy is the state this pipeline last wrote, so falling back to it never mutates a foreign artifact; the invariant survives without the wedge. etag → digest: an etag is host-assigned and survives neither recompression nor a CDN swap. The criterion was written before the thing was built, and the build falsified it.
• AC-4: the six curated files still read from the checkout, unchanged. A spec pins that this ticket did not move them, because they are the irreplaceable half and the derived half is the only half that may leave git.
• AC-5: the artifact URL resolves from one declared config value with no host string re-derived at any use site.

Out of Scope

• Publishing without committing. That is Axis 2 (#17238) and the ticket where growth actually stops. This one deliberately keeps committing, so it is reversible and its output is diffable against today's.
• Relocating DevIndex. Same axis, same ticket.
• Deleting anything from git history. Axis 3.
• neomjs/pages. Unaffected here.

Avoided Traps

• Implementing the fetch in the workflow or in dataSyncPipeline.mjs. It reads correctly and is thrown away the moment DevIndex relocates. The service layer is the surface that travels.
• Falling back silently. A quiet fallback makes this ticket appear complete while the pipeline still depends on the checkout, and nothing would surface it until Axis 2 fails for reasons that look unrelated.
• Treating a short read as a smaller index. Without the etag assertion, a truncated fetch and a genuinely shrunken index are indistinguishable, and the pipeline would happily publish the truncation. This is the same silent-wrong-answer class that produced the defect in the first place.

Related

Parent: #17238 · siblings: Axis 2 and Axis 3 of the same epic.

Retrieval Hint: query_raw_memories("devindex published artifact SSOT etag storage read side") · "derived versus curated 26.5 MiB versus 543 bytes"

Live latest-open sweep: checked the latest 20 open issues at 2026-08-19T08:36:42Z plus a 30-message A2A claim scan across all read-states; no equivalent ticket and no in-flight claim on this scope.

Origin Session ID: a105d215-c261-4b34-82a9-546596f665ef