Cloud tenant-repo CLI sync: 3 Day-0 substrate gaps

Context

First-real-world cloud-deployment exercise (R3 first-tenant ingestion smoke, post-#12030 RawRepoSource merge) triggered three discrete substrate gaps that block the operator-bulk CLI path node ai/scripts/maintenance/syncTenantRepos.mjs --repo-slug <slug>. Each is independently mechanical but stacks into a Day-0 unbootable state for cloud-deploy tenant ingestion.

All three surfaced inside a single sync attempt — fixed one, hit the next. Filing as one ticket because they all gate the same CLI path; PR can address in distinct commits.

Bug A — Runtime image missing git binary

Symptom

[ERROR] [TenantRepoSync] <tenant-id>/<repo-slug> failed:
KB_TENANT_REPO_SYNC_SYNC_FAILED (GitMirror clone failed)

Underlying (from orchestrator stderr):

[Backup] failed to capture gitSha: spawn git ENOENT

Root cause

ai/deploy/Dockerfile:21-26 runtime stage installs only libstdc++:

FROM node:24-alpine
WORKDIR /app
RUN apk add --no-cache libstdc++
COPY --from=builder /app ./

But ai/services/knowledge-base/helpers/GitMirror.mjs shells out to git clone / git fetch. The node:24-alpine base image doesn't include git.

Fix

Add git to the runtime-stage apk install:

RUN apk add --no-cache libstdc++ git

Image size delta is ~23 MiB (verified by in-container apk add test).

Bug B — TenantRepoSyncService.resolveIngestionService looks up wrong export name

Symptom

After Bug A fixed (clone succeeds), next failure:

[ERROR] [TenantRepoSync] <tenant-id>/<repo-slug> failed:
KB_TENANT_REPO_SYNC_SYNC_FAILED (Cannot read properties of undefined (reading 'ingestSourceFiles'))

Root cause

ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:564-567:

async resolveIngestionService() {
    const services = await import('../../../services.mjs');
    return services.KB_KnowledgeBaseIngestionService || services.KnowledgeBaseIngestionService;
}

But ai/services.mjs:212 and ai/services.mjs:282 export the service as KB_IngestionService:

const KB_IngestionService = makeSafe(_KB_IngestionService, kbSpec);
// ...
export {
    // ...
    KB_IngestionService,
    // ...
};

Neither KB_KnowledgeBaseIngestionService nor a bare KnowledgeBaseIngestionService is exported anywhere — resolveIngestionService always returns undefined, then the ingestionService.ingestSourceFiles(...) call NPEs.

Architectural sub-question

The orchestrator's in-process invocation (viaMcp: false hardcoded at line 442) implies a same-process deployment model where the orchestrator + kb-server share runtime. In the canonical cloud topology (ADR 0014) they're separate containers; the orchestrator's KB_IngestionService instance would connect to the shared chroma + sqlite, but is otherwise an independent process from the kb-server's KB_IngestionService instance. Is in-process the intended path, or should the orchestrator route through the kb-server's MCP ingestSourceFiles tool? Worth answering before patching the export name, because the export-name fix unblocks the in-process path which may or may not be the right architecture.

Fix (assuming in-process is correct)

async resolveIngestionService() {
    const services = await import('../../../services.mjs');
    return services.KB_IngestionService;
}

Drop the || fallbacks; the resolver should error loudly if the canonical export is renamed in the future, not silently fall through to undefined.

Bug C — NEO_TENANT_REPO_MIRROR_ROOT env var is dead config

Symptom

After Bug B fixed (in test container), every tenant repo entry must explicitly set mirrorRoot per-repo. The canonical ai/deploy/docker-compose.yml declares NEO_TENANT_REPO_MIRROR_ROOT=/app/.neo-ai-data/tenant-repos on orchestrator's environment: block — but nothing in the code reads it.

Root cause

ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:419,426,433 reads repo.mirrorRoot per-entry only:

await gitMirror.cloneIfMissing({
    tenantId      : repo.tenantId,
    repoSlug      : repo.repoSlug,
    mirrorRoot    : repo.mirrorRoot,
    // ...
});

No fallback to process.env.NEO_TENANT_REPO_MIRROR_ROOT or to an aiConfig.tenantRepoMirrorRoot global default.

Additional shape mismatch

The canonical compose env value (/app/.neo-ai-data/tenant-repos) matches the volume mount point, but deriveTenantRepoMirrorPath (TenantRepoAccessContract.mjs:279) appends tenant-repos itself:

return path.join(root, 'tenant-repos', tenantSegment, ...repoSegments);

So if the env var were wired up with the current value, paths would double: /app/.neo-ai-data/tenant-repos/tenant-repos/<tenant>/<repo>. The correct env value would be /app/.neo-ai-data.

Fix

Two-part:

1. Wire the env var up as a global default in resolveTenantReposConfig (or wherever per-repo mirrorRoot is materialized). Per-repo mirrorRoot overrides; absence falls back to process.env.NEO_TENANT_REPO_MIRROR_ROOT then to a hardcoded /app/.neo-ai-data (matching the canonical-compose volume layout).
2. Fix the canonical compose env value to /app/.neo-ai-data (drop the trailing tenant-repos segment) — OR — adjust deriveTenantRepoMirrorPath to not append tenant-repos. The first option preserves backward compat for any existing mirrorRoot: per-repo overrides; the second tightens the helper. Decide during PR design review.

The Architectural Reality

• ai/deploy/Dockerfile — runtime stage missing git apk install (Bug A).
• ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:564-567 — wrong export name lookup (Bug B).
• ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:419-433 — no env-var fallback for mirrorRoot (Bug C, part 1).
• ai/deploy/docker-compose.yml — orchestrator env block has dead NEO_TENANT_REPO_MIRROR_ROOT (Bug C, part 2).
• ai/services/knowledge-base/helpers/TenantRepoAccessContract.mjs:279 — path-builder appends tenant-repos (Bug C, part 2 cont.).

Decision Record impact

aligned-with ADR 0014 — extends cloud-deployment substrate within the existing topology without redrawing service boundaries.

Acceptance Criteria

• Bug A: ai/deploy/Dockerfile runtime stage adds git to apk install.
• Bug B: TenantRepoSyncService.resolveIngestionService returns services.KB_IngestionService (no fallback chain to silently-undefined names).
• Bug B follow-up: design decision documented in PR body — is in-process (viaMcp: false) the correct orchestrator→ingestion path, or should it route via kb-server's MCP tool? Either way, the answer is recorded.
• Bug C: mirrorRoot falls back to process.env.NEO_TENANT_REPO_MIRROR_ROOT then to a aiConfig.tenantRepoMirrorRoot default; per-repo overrides preserved.
• Bug C cont.: canonical compose's NEO_TENANT_REPO_MIRROR_ROOT value is consistent with the path-builder's expected root (either drop the helper's 'tenant-repos' append, or fix the env value to the parent dir).
• First-time clean docker compose up -d --build followed by docker compose exec orchestrator node ai/scripts/maintenance/syncTenantRepos.mjs --repo-slug <slug> succeeds end-to-end without operator-side intervention (no apk add git, no per-repo mirrorRoot override required for the canonical deployment shape, no in-container source patches).

Out of Scope

• Architectural redesign of orchestrator-ingestion topology (in-process vs MCP-call) — record the answer, don't rewrite. If the consensus is MCP-call, that's a separate ticket for that work.
• Multi-tenant credential scaling (separate ticket #12032 already filed).
• New parser registration / KB content concerns — this ticket is strictly about the orchestrator-bulk CLI path.

Avoided Traps

• Filing three tickets: would have 3× the ticket-create overhead for problems all blocking the same CLI path. One umbrella with distinct ACs lets the PR review batch them.
• In-container monkey-patching: tempting for verification continuity, but masks the substrate gaps from canonical rebuilds. The Day-0 promise of the cloud-deployment substrate is "clone, build, sync, ingest" with no operator-side surgery.

Related

• Parent epic: #11731 (Server-side tenant-repo ingestion)
• Substrate foundation: #11787 (config/credential contract), #11788 (GitMirror), #11789 (envelope builder), #11790 (sync lane)
• Sibling: #12032 (credentialRef file: scheme — credential scaling)
• Empirical anchor: first cloud-deployment tenant-ingestion attempt, 2026-05-26