Private tenant repos cannot be read: the blob fetch has no credential

Context

Found while diagnosing an external Agent OS deployment whose tenant repos are private and hosted on a self-managed Git server. Its four tenant repos clone successfully, list successfully, and then every one of them fails at the same stage with KB_INGEST_ENVELOPE_FILE_READ_FAILED. The KB document count has never left zero.

Observation vs inference, kept separate: the deployment's own failure boundary (clone OK, list OK, read fails) is observed from its deployment-state snapshot. The mechanism below is not inferred from that — it is reproduced locally against a private remote, receipt in The Problem. The underlying git stderr on that deployment has not been read yet; its orchestrator log is not currently reachable. The local reproduction is the evidence this ticket rests on.

The Problem

#16546 / PR #16547 made the tenant mirror blobless (--mirror --filter=blob:none). That was correct and measured — 4.9 GB → 36.2 MB, and it ended an orchestrator OOM loop. gitMirror.mjs states the trade in its own JSDoc at :904-907:

the single content read, show <revision>:<path> … becomes a potentially NETWORKED read. The lane is already a network operation behind the same credential so this adds no new failure domain

The premise in that last clause does not hold. The lane is a network operation behind a credential, but show is not — it is the one git invocation that never receives one.

credentialRef reaches runGit at exactly two of its eleven call sites:

Credentials cannot leak across invocations either: createGitExecutionEnvironment (:390) builds a disposable mkdtemp HOME with an empty .gitconfig per call and deletes it after, delivering the secret through GIT_ASKPASS (:365-366). Nothing is written into the mirror's own config, by design. So an invocation without credentialRef is genuinely unauthenticated, not incidentally so.

Reproduction — a private remote, the three calls the code makes

Blobless-cloned a private repository with a credential, exactly as cloneIfMissing does, then ran the two read paths under a disposable HOME with an empty .gitconfig:

clone --mirror --filter=blob:none  (with credential)   → exit 0, promisor=true, partialclonefilter=blob:none

ls-tree -r --name-only <rev>       (no credential)     → exit 0   ← listRevisionPaths, gitMirror.mjs:1181
show <rev>:<path>                  (no credential)     → exit 128 ← readRevisionFile,   gitMirror.mjs:1210
    fatal: could not read Username for 'https://<private-git-host>': terminal prompts disabled
    fatal: could not fetch 1a0cacca1cafd0f2ebc7b35020fb5867fefd3620 from promisor remote

show <rev>:<path>                  (with credential)   → exit 0

Same mirror, same revision, same path, one variable. The failure boundary reproduces the deployment's exactly: listing succeeds, reading fails.

Why no test or plane has ever caught it

The only tenant repo this project has ever ingested is public. #16557's measurements have show lazily fetching perfectly well on this plane — 0.42 s/file, mirror backfilling 36.2 MB → 121.9 MB as it read. A public remote serves a promisor fetch to an anonymous client, so the missing credential is unobservable. The defect is invisible in exactly the visibility class we test in, and fires on every repository in the class we do not.

That also makes the blast radius the whole feature: server-side tenant-repo ingestion exists to ingest tenants' repositories, which are private by default.

The Architectural Reality

• ai/services/knowledge-base/helpers/gitMirror.mjs:1200 — readRevisionFile destructures {mirrorRoot, tenantId, repoSlug, revision, sourcePath}. There is no credentialRef in the signature, so a caller cannot supply one even by accident.
• ai/services/knowledge-base/helpers/tenantRepoIngestEnvelopeBuilder.mjs:222 — calls gitMirror.readRevisionFile({...identity, revision, sourcePath}). identity is spread into every mirror call, so it is the natural carrier; today it does not hold a credential and the callee would discard one.
• ai/services/knowledge-base/helpers/gitMirror.mjs:390 / :365 — per-invocation disposable HOME + GIT_ASKPASS. This is why the credential must be threaded rather than assumed ambient, and it is correct as written.
• resolveHead (:1072), isAncestor (:1102) and diffRevisions (:1122) also receive no credential and need none — refs and --name-status never read blob content. The gap is specific to the one content read, not general.

The Fix

1. Add credentialRef to readRevisionFile's options and forward it to runGit, alongside knownHostsPath as cloneIfMissing/fetch already do.
2. Thread the tenant's credentialRef into the identity object the envelope builder spreads, so buildFilePayloads → readRevisionFile carries it per repo.
3. Do not blanket-add credentialRef to the remaining call sites. Reads that cannot touch the network should stay unable to resolve a secret; widening that is a security regression disguised as consistency.

Contract Ledger

Decision Record impact

none. This restores the credential invariant #16546 assumed rather than changing any decision.

Acceptance Criteria

• A spec fails on today's code and passes after the fix: a blobless mirror of a repository whose remote requires authentication, where readRevisionFile succeeds only when a credential is threaded. The remote must actually reject anonymous fetches — a fixture reachable without credentials cannot fail on this defect and tests nothing.
• listRevisionPaths is asserted to still succeed without a credential in the same spec, pinning that the fix does not paper over the boundary by making everything authenticated.
• The call sites that must not gain credentialRef (:1072, :1102, :1122, :1181) are asserted to still operate credential-free, so a later "consistency" refactor cannot widen secret resolution silently.
• The :885 JSDoc's trade paragraph is corrected: it currently asserts the read is "behind the same credential", which is the false premise this ticket fixes.
• A full envelope builds against a private tenant repository, measured on a deployment rather than a fixture. (Post-merge; requires a deployment with a private tenant.)

Out of Scope

• #16557 — the cost of those lazy fetches (23,931 round trips). It stays held on the declaration model. This ticket is underneath it: on a private tenant the fetches do not merely cost, they fail. Fixing this does not fix that, and fixing that does not fix this.
• #16566 — the embed-stage blocker. A repository that gets past the read still meets it.
• Whether the mirror should be blobless at all. #16546's disk and heap results are measured and stand.
• The unexplained fourth repository on that deployment: three fail at read, one reached the embed stage. Most plausibly it has nothing to read, but that is not established and this ticket does not claim it.

Avoided Traps

• Reading the clause and not the code below it. The :904 JSDoc says the trade "adds no new failure domain … behind the same credential". That sentence is why this is easy to walk past: the risk is documented, so it reads as handled. The credential it names is never passed. A documented trade is not a discharged one.
• Probing with a remote that cannot produce the failure. Any public remote — including this project's own — serves the promisor fetch anonymously and returns exit 0. A green result there is not evidence about this defect; the reproduction has to run against a remote that genuinely refuses anonymous access.
• Blaming the mirror root. The competing hypothesis was a per-repo mirrorRoot disagreeing with its environment default. KB_INGEST_ENVELOPE_LIST_FAILED (tenantRepoIngestEnvelopeBuilder.mjs:191) is what a wrong root produces — listing would fail first. The deployment reports the file-read code, which places the fault after a successful list and rules the root out without needing the log.

Related

#16546 / PR #16547 (introduced the blobless mirror this depends on) · #16557 (the cost half of the same trade; Vega, held) · #16566 (embed-stage blocker, parent of #16557) · #16569 (plane-id default — same deployment).

Live latest-open sweep: checked the latest 20 open issues at 2026-08-07T13:09:29Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages: no competing lane-claim on this scope.

Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2

Retrieval Hint: query_raw_memories("blobless promisor mirror git show unauthenticated lazy blob fetch private tenant repo credentialRef readRevisionFile")

Retrieval Hint: the discriminating measurement is ls-tree vs show on the same blobless mirror of a private remote, under a disposable HOME with an empty .gitconfig — exit 0 vs exit 128 could not fetch … from promisor remote.