Blobless tenant mirror turns first ingestion into 23,931 network round trips

Status update 2026-08-06 — the live mirror is now fully backfilled; this no longer blocks ingestion here

Authoritative census, run against the mirror volume mounted read-only (no live state mutated):

git rev-list --objects --missing=print HEAD --max-count=1 | grep '^?'

missing blobs at HEAD tree : 0
total objects in HEAD tree : 28374
mirror size                : 121.9M   (was 36.2M immediately post-#16547)
promisor: true   partialclonefilter: blob:none   tracked files at dev: 23952

Every blob the HEAD tree references is local. The 36.2M → 121.9M growth is the lazy backfill: the three failed TenantRepoSync attempts (consecutiveFailures=3) each walked the tree and fetched as they read. So the ~2.8h first-ingest cost has already been paid on this plane — and paid for nothing, since every one of those runs then died at the embed stage (KB_VECTOR_EMBED_FAILED, #16566).

Read-path measurement, 20 sequential git show HEAD:<path>:

assertBloblessClone's guarantee is intact — promisor=true and partialclonefilter=blob:none are both still set; the growth is lazy backfill, not a filter regression. The 4.9 GB has not returned.

Disposition: keep open, DEPRIORITISE below #16566. The defect is real for any fresh mirror — a new tenant still pays N round trips on first ingest — but it is not currently blocking anything. Tenant ingestion's live blocker is entirely the embed stage. And for the immediate goal, a small first tenant is cheap even on a cold mirror, because cost scales with file count.

Option A is still untested, and a second invalid attempt is recorded below so it is not repeated a third time.

Context

Found on the live plane immediately after deploying #16547 (#16546) and removing the pre-fix 4.9 GB mirror, while verifying the fix did what it claimed.

The disk half worked exactly as designed: 4.9 GB → 36.2 MB, remote.origin.promisor=true, partialclonefilter=blob:none, 28 promisor packs, 25,839 commits, 23,931 files at dev, tip at the deployed revision. Orchestrator memory went from ~1009 MB at death to 146 MB / 3 GB, and the OOM loop stopped — 19-for-19 heap FATALs before, zero since.

Then the operator asked the question none of the three reviewers asked: "resources/content alone is 126 MB. OC does not need it, but for tenant repo ingestion, it is crucial."

The Problem

A blobless mirror makes every content read a network round trip, and the ingestion does one show per file.

gitMirror.mjs reads content exactly one way — show <revision>:<sourcePath> — which on a promisor repo lazily fetches that single blob from the remote.

Measured on the live plane, two independent samples:

The mirror grew 36.2 MB → 39.8 MB across those fetches, which is the backfill confirming each read hit the network.

Tracked files at dev: 23,931. With lastIngestedRev null the first run reads the whole tree, not a diff: 23,931 × 0.42 s ≈ 2.8 hours, of which resources/content (16,674 files) is ≈ 1.9 hours.

This is the trade #16546 explicitly rejected, arriving through a different door. Its Avoided Traps rejected --depth 1 because it "trades a disk problem for a throughput problem"; --filter=blob:none preserves the incremental-diff operations --depth would have broken but reintroduces the throughput cost on the read path instead. The PR body states the trade and prices it as a failure-mode change; nobody priced it as per-file latency multiplied by the ingestion surface.

The Architectural Reality

• ai/services/knowledge-base/helpers/gitMirror.mjs — readRevisionFile is the only content read, one show per path. On a promisor repo each is a round trip.
• cloneIfMissing — --mirror --filter=blob:none, with assertBloblessClone proving blobs are genuinely absent. Working as intended; the cost is downstream.
• The incremental path is not affected in steady state: once lastIngestedRev is set, only changed paths are read. The 2.8 hours is a first-ingest / re-clone cost.

The Fix

Direction, not prescription.

Option A — bulk-fetch before ingesting. Git can fetch many missing objects in one negotiation; the ingestion never asks for them together. A pre-ingest pass resolving the revision's blob OIDs and fetching them in one operation would collapse N round trips into one. Still untested — see the two invalid attempts below.

Option B — filter by path, not by blob. --filter=blob:none is the wrong axis if the ingestion reads most of the current tree anyway. --filter=sparse:oid or a partial clone scoped to the ingested subtree keeps the current tree's blobs local while still dropping history.

Option C — accept it and pay the first ingest once. Defensible if first-ingest is genuinely rare; note this plane has now paid it involuntarily, via failed runs.

Acceptance Criteria

• The per-file cold-read cost is measured and recorded (0.42 s/file cold; ≪ that warm) so a threshold assertion can be written against a real number.
• The measurement method is recorded — see Avoided Traps; the naive instruments are wrong in three distinct ways.
• assertBloblessClone's guarantee verified intact after backfill; the 4.9 GB has not returned.
• A full first ingestion completes within a stated budget, measured on the container plane rather than a fixture. Blocked on #16566 — no ingestion completes at all today.
• The per-file cold-read cost is asserted against a threshold that fails at 0.42 s/file, so the regression cannot return silently. Requires a genuinely cold mirror, which this plane can no longer produce.
• Option A tested properly, on a fresh cold clone.
• Incremental sync stays untouched: diff --name-status <base> <head> and merge-base --is-ancestor still operate without fetching blobs.

Out of Scope

• #16554 (unowned lanes) — different producers; neither substitutes.
• The orchestrator heap ceiling (#16463).
• Reverting #16547. The disk and OOM results are real and measured; this is the other half of the same trade.

Avoided Traps

• Trusting a container-side timer. date +%s%N under busybox produced "20 files: 0 ms, 6.8 MB" — a result that looked like a triumph and was a broken clock. Timed from the host thereafter. (date +%s at second granularity does work.)
• Reporting a failed instrument as a negative result. The original batch-vs-sequential comparison returned 12.42 s vs 12.55 s, reading as "batching does not help." It fell through to sequential in both arms — fetch-pack --stdin failed. A test that did not run is not evidence.
• Second instance of the same trap, 2026-08-06. A retest searched for locally-missing blobs to obtain a cold sample, found none in range, and both arms consequently timed warm reads and returned 0s. Again not evidence about batching. Testing Option A now requires a deliberately fresh cold clone.
• Probing absence with a command that fetches the object under test. ls-tree --long fetches blobs to report sizes; under a writable mount cat-file -e triggers the promisor fetch it is testing for. The fetch-free APIs are git rev-list --objects --missing=print and GIT_NO_LAZY_FETCH=1.
• Declaring the fix verified on the axis that was easy to measure. Size, promisor config, byte-identical content and zero OOMs were confirmed and called done — without measuring the throughput of the thing the mirror exists to serve.

Related

#16546 / PR #16547 (the merged change this qualifies) · #16566 (the embed-stage blocker that gates this ticket's remaining ACs) · #16551 (backoff / frozen counters) · #16554 · #16463.

Origin Session ID: 8921d480-6087-4bfa-abe0-4f47873e06c4

Retrieval Hint: query_raw_memories("blobless tenant mirror lazy blob fetch per-file network round trip first ingestion 23931 files 0.42s")

Retrieval Hint: the discriminating measurement is N sequential git show <rev>:<path> on a promisor repo, timed from the HOST, against the mirror's own size growth as the backfill receipt.