Context
ai/services/knowledge-base/helpers/gitMirror.mjs:904 clones a tenant repo as:
await runGit(['clone', '--mirror', cleanCloneUrl, mirrorPath], {…});--mirror is the maximal form: every ref, every commit, and every blob in history.
Measured on the live plane just now:
| fact |
value |
/app/.neo-ai-data/tenant-repos/neo-shared |
4.9 GB |
| shallow marker present |
no — full clone |
orchestrator NODE_OPTIONS |
empty |
| Node default heap ceiling in that container |
1728 MB |
| observed death |
FATAL ERROR: Reached heap limit at ~1009 MB, allocation failure; scavenge might not succeed |
| container memory limit |
3 GB |
The orchestrator dies roughly every 290 s with tenant repo sync (cloud) active. Until #16517 landed, the self-succession lease defect amplified each death into a ~36 s restart cycle (RestartCount reached 1528); with the lease fixed the cadence is ~5 minutes, so the loop is quieter, not gone — which is worse for detection.
The Problem
The mirror pays for history the ingestion never reads.
V-B-A on what actually consumes the mirror — every runGit invocation in gitMirror.mjs:
| call |
needs |
rev-parse --git-dir |
repo metadata |
for-each-ref |
refs |
rev-parse --verify <ref>^{commit} |
commits |
merge-base --is-ancestor |
commit graph |
diff --name-status -z -M <base> <head> |
two trees |
ls-tree -r -z --name-only <revision> |
one tree, names only |
show <revision>:<sourcePath> |
one blob, only for paths actually ingested |
So only the latest version of the files that get ingested is ever read as content. Historical blobs — the overwhelming majority of a 4.9 GB repo — are downloaded, stored, and never opened.
This scales the wrong way in exactly the deployment that needs it most: a polling multi-repo cloud deployment holds one full mirror per tenant repo, so disk and sync cost grow as N repos × full history while the consumed surface stays N × current tree.
Why --depth is the wrong fix
The obvious shallow clone breaks incremental sync. merge-base --is-ancestor and diff --name-status <base> <head> both need the base revision reachable — that is how the lane computes "what changed since last time". A depth-1 clone makes the previous revision unreachable and forces every sync to re-ingest the whole tree, trading a disk problem for a throughput problem.
The Fix
--filter=blob:none — a blobless partial clone. It is precisely shaped to the table above:
- commits and trees are still complete, so
for-each-ref, rev-parse, merge-base --is-ancestor and diff --name-status all keep working unchanged;
- no blobs are downloaded at clone time;
show <revision>:<path> fetches exactly the blobs it asks for, lazily, and only for paths the ingestion consumes.
Git records the filter in the clone's config (remote.origin.promisor, remote.origin.partialclonefilter), so the existing fetch --all --prune inherits it without a second change.
Known trade, stated rather than discovered later: a blobless clone makes show a potentially networked operation. The lane is already a network operation and runs behind the same credential, so this does not add a failure domain — but a show against an unreachable remote now fails where it previously read from disk. That belongs in the code comment at the call site.
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback / Error Semantics |
Docs |
Evidence |
cloneIfMissing clone form |
this ticket |
clone --mirror --filter=blob:none — bare mirror, no historical blobs |
Server without partial-clone support ⇒ clone fails with the existing KB_GITMIRROR_CLONE_FAILED; it must NOT silently fall back to a full clone, or the regression returns invisibly |
call-site comment |
a clone against a fixture remote reports a promisor config |
| blob availability |
git partial-clone |
show <rev>:<path> lazily fetches the one blob |
Unreachable remote ⇒ show fails loudly, never returns empty content |
call-site comment |
show on a blobless clone returns the file's real bytes |
| incremental diff |
existing lane |
unchanged — merge-base / diff --name-status operate on trees |
n/a |
— |
the diff between two revisions is identical on a blobless clone |
Decision Record impact
none — a clone-flag change behind an existing helper boundary. No new contract.
Acceptance Criteria
Out of Scope
- The orchestrator heap ceiling. #16463 owns whether the ceiling is right and whether ~500 MB is a leak; the ceiling is separately questionable for multi-repo ingestion and does not belong in a clone-flag change.
- Re-cloning existing mirrors. An already-full mirror stays full and keeps working; migration of existing mirrors is a follow-up if the disk matters more than the churn.
- The KB corpus loss. Under separate investigation; this ticket does not claim to cause or fix it.
Avoided Traps
--depth 1. Breaks merge-base --is-ancestor and the base-to-head diff, forcing full re-ingestion every sync.- A silent full-clone fallback. Would make the fix unobservable the moment a remote lacked partial-clone support, and the 4.9 GB would come back with green tests.
- Fixing the symptom by raising the heap. The failure signature is
allocation failure; scavenge might not succeed — a large contiguous allocation, so a higher ceiling moves the wall rather than removing it.
Intake note
Abbreviated intake, disclosed rather than implied: live open-queue sweep run against the current backlog (tenant mirror clone size, gitMirror, blob filter, shallow, orchestrator memory/heap/OOM) found no parallel ticket; #16463 is the adjacent heap-ceiling lane and is deliberately kept separate. Operator directive was explicit that this phase needs fixes rather than further instrumentation.
Origin Session ID: c724a85f-2d37-44ac-9a33-12dcce415aa2
Retrieval Hint: query_raw_memories("tenant mirror blobless partial clone filter blob none orchestrator OOM 4.9GB full history ingestion reads only current tree")
Context
ai/services/knowledge-base/helpers/gitMirror.mjs:904clones a tenant repo as:await runGit(['clone', '--mirror', cleanCloneUrl, mirrorPath], {…});--mirroris the maximal form: every ref, every commit, and every blob in history.Measured on the live plane just now:
/app/.neo-ai-data/tenant-repos/neo-sharedNODE_OPTIONSFATAL ERROR: Reached heap limitat ~1009 MB,allocation failure; scavenge might not succeedThe orchestrator dies roughly every 290 s with
tenant repo sync (cloud)active. Until #16517 landed, the self-succession lease defect amplified each death into a ~36 s restart cycle (RestartCountreached 1528); with the lease fixed the cadence is ~5 minutes, so the loop is quieter, not gone — which is worse for detection.The Problem
The mirror pays for history the ingestion never reads.
V-B-A on what actually consumes the mirror — every
runGitinvocation ingitMirror.mjs:rev-parse --git-dirfor-each-refrev-parse --verify <ref>^{commit}merge-base --is-ancestordiff --name-status -z -M <base> <head>ls-tree -r -z --name-only <revision>show <revision>:<sourcePath>So only the latest version of the files that get ingested is ever read as content. Historical blobs — the overwhelming majority of a 4.9 GB repo — are downloaded, stored, and never opened.
This scales the wrong way in exactly the deployment that needs it most: a polling multi-repo cloud deployment holds one full mirror per tenant repo, so disk and sync cost grow as N repos × full history while the consumed surface stays N × current tree.
Why
--depthis the wrong fixThe obvious shallow clone breaks incremental sync.
merge-base --is-ancestoranddiff --name-status <base> <head>both need the base revision reachable — that is how the lane computes "what changed since last time". A depth-1 clone makes the previous revision unreachable and forces every sync to re-ingest the whole tree, trading a disk problem for a throughput problem.The Fix
--filter=blob:none— a blobless partial clone. It is precisely shaped to the table above:for-each-ref,rev-parse,merge-base --is-ancestoranddiff --name-statusall keep working unchanged;show <revision>:<path>fetches exactly the blobs it asks for, lazily, and only for paths the ingestion consumes.Git records the filter in the clone's config (
remote.origin.promisor,remote.origin.partialclonefilter), so the existingfetch --all --pruneinherits it without a second change.Known trade, stated rather than discovered later: a blobless clone makes
showa potentially networked operation. The lane is already a network operation and runs behind the same credential, so this does not add a failure domain — but ashowagainst an unreachable remote now fails where it previously read from disk. That belongs in the code comment at the call site.Contract Ledger Matrix
cloneIfMissingclone formclone --mirror --filter=blob:none— bare mirror, no historical blobsKB_GITMIRROR_CLONE_FAILED; it must NOT silently fall back to a full clone, or the regression returns invisiblyshow <rev>:<path>lazily fetches the one blobshowfails loudly, never returns empty contentshowon a blobless clone returns the file's real bytesmerge-base/diff --name-statusoperate on treesDecision Record impact
none— a clone-flag change behind an existing helper boundary. No new contract.Acceptance Criteria
cloneIfMissingclones with--filter=blob:nonealongside--mirror.diff --name-status <base> <head>between two revisions returns the same result on a blobless clone as on a full one, so incremental sync is provably unaffected.show <revision>:<path>returns the file's real content on a blobless clone — the lazy fetch is exercised, not assumed.KB_GITMIRROR_CLONE_FAILEDand does not fall back to a full clone.showtrade.Out of Scope
Avoided Traps
--depth 1. Breaksmerge-base --is-ancestorand the base-to-head diff, forcing full re-ingestion every sync.allocation failure; scavenge might not succeed— a large contiguous allocation, so a higher ceiling moves the wall rather than removing it.Intake note
Abbreviated intake, disclosed rather than implied: live open-queue sweep run against the current backlog (
tenant mirror clone size,gitMirror,blob filter,shallow,orchestrator memory/heap/OOM) found no parallel ticket; #16463 is the adjacent heap-ceiling lane and is deliberately kept separate. Operator directive was explicit that this phase needs fixes rather than further instrumentation.Origin Session ID: c724a85f-2d37-44ac-9a33-12dcce415aa2
Retrieval Hint:
query_raw_memories("tenant mirror blobless partial clone filter blob none orchestrator OOM 4.9GB full history ingestion reads only current tree")