Cloud-Native KB Ingestion — Migration Path
Status — post-split migration. The Phase 0/1 zero-config promise below is historical context. The current shared-core scan is a deliberate Engine/Brain repository-profile cut; tenant-config persistence remains documented in Configuration.
Historical Phase 0/1 zero-config promise
The original cloud-ingestion substrate let a single-repo Neo deployment adopt tenant plumbing without a config edit. That claim applied to its original SourceRegistry-backed kbSync path; later repository splits and the github-content-sync tenant changed the source set. Do not use the old byte-equivalence claim as current migration evidence.
Current shared-core boundary
| Concern | Current behavior |
|---|---|
| Shared core acquisition | kbSync reads the image-carried Engine package and Brain source through two exact-revision profiles. It no longer enumerates SourceRegistry or reads aiConfig.sourcePaths for the core scan. |
| Shared core ownership | The two outputs are stamped separately as (neo-shared, neo) and (neo-shared, neo-agent-brain); one combined JSONL cannot represent both safely. |
| Conversations | The old three Engine-tree Sources retired in #402. ConversationCorpusSource is a distinct github-content-sync tenant route; its activation and freshness receipts are #411. |
| First profile embed | Additive (deleteStale: false). The post-cut legacy scan remains disabled; old and new IDs for one code fact can coexist until #419's replacement-proven retirement. Old conversation rows have their separate #417 owner. A corpus-tenant receipt alone never authorizes legacy kbSync stale deletion. |
The first profile materialization publishes two JSONL artifacts behind one manifest, with separate revision and extraction identities. The current code does not claim byte-equivalence with the retired global SourceRegistry builder. Release-note files and the skills corpus have no active image-carried roots in this cut; their separately owned source contracts remain outside the shared-core profiles.
Activation and legacy-row migration receipt
This is the acceptance plan for #282 AC-9, not an activation performed by its code PR. kbSync remains off under #253 / #411 until a later activation owner records each step against the same deployed image and collection:
- Capture the pre-activation Chroma row inventory by
(tenantId, repoSlug, sourcePath, type)and count old code, conversation, skill, concept, and release-note rows separately. Bind the installed Engine pin, Brain revision, and current corpus revision in the receipt; an aggregate count cannot prove which family was preserved. - Resolve effective
tenantReposthrough the graph → YAML → Tier-1 winner. The core scan and tenant GitMirror both refuse an enabled shared-tenant Engine/Brain route by owner key or canonical clone URL before writes. A higher-tiertenantRepos: []suppresses lower-tier entries; reading Tier-1 alone is not an ownership check. - Materialize both exact-revision profiles and record their manifest SHA, extraction identity, per-family yielded paths and counts. The full profile embed is additive (
deleteStale: false): it can leave a legacy and a new ID for the same Engine code fact because extraction identity changes the hash. Until #419's deployed cleanup receipt, neither a green embed norask_knowledge_basecan prove code currentness. Verify both repository stamps, same-path collision separation, and stored-content hydration against representative old and new rows. No global collectionshadow-swapor per-repo stale deletion is authorized by this receipt. - Retire old conversation rows only through #417's conversation-scoped control. #419 owns a separate replacement-proven source-code-row retirement: it compares the exact old IDs against landed Engine/Brain profile rows and preserves unmatched families. Neither transaction may delete rows of another family as a side effect.
- The Skills package is a separate repository: the Brain image's
.agents/skillslink is not a revision-bound Brain input. Name and verify its ingestion owner before retiring oldSkillSourcerows. The #282 Engine pin17b59aad8f95c55c916fd6bb8bd6a0f43bd2d687contains 59resources/content/conceptsfiles; the Engine content-removal lane must preserve a declared, revision-bound concept input before deleting that directory. Release-note roots are absent from both images in this cut; compare any retained historical rows before deciding their retirement.
Only after these receipts pass may an activation owner change the kbSync gate and choose a scoped stale-row strategy. A corpus tenant receipt alone cannot make the shared core scan current.
What an operator opts into (cloud / multi-tenant mode)
Divergence from the single-repo default is granular and opt-in. An operator moving to a multi-tenant cloud deployment changes only what their topology requires:
- Skip legacy default Source registration — set
aiConfig.useDefaultSources = falsefor SourceRegistry consumers. This does not disable the shared Engine/Brain core profiles;kbSynchas its own scheduler toggle. - Unknown tenant repo shape — set
aiConfig.rawRepoSource = trueto register the built-in raw-text fallback Source while a custom Source is still premature. - Different legacy Source layout — override affected
aiConfig.sourcePathskeys for a consumer still using that registry. Repository profiles carry their own route territories and exact reader authority;sourcePathsdoes not override them. - Register tenant Sources/Parsers — populate
aiConfig.customSources/aiConfig.customParserswith pre-imported tenant classes, or callSourceRegistry.registerSource(...)at runtime. - Spoof-rejection policy — a multi-tenant operator should consider
aiConfig.spoofRejectionMode: 'reject'(fail-closed) over the'overwrite'default (see Security).
Each of these is a local config edit; none requires a code fork.
Config-template clone-sync
The new config keys (useDefaultSources, rawRepoSource, useDefaultParsers, customSources, customParsers, sourcePaths, defaultTenantId, defaultRepoSlug, defaultVisibility, spoofRejectionMode) live in ai/mcp/server/knowledge-base/config.template.mjs — the SourceRegistry keys via PR #11659, the sourcePaths keys via PR #11661, and the tenant-stamping keys via PR #11662. Each clone's local config.mjs is gitignored and copied from the template.
- Zero-config deployments need no local
config.mjsedit — the runtime falls through to defaults when a key is absent. - Cloud / tenant-mode deployments add the keys they need to their local
config.mjs. A harness restart picks up the change (config modules load once per MCP process).
Tenant-config persistence
How a multi-tenant deployment stores its per-tenant configuration — the KnowledgeBaseTenantConfig graph-node shape, the kb-config.yaml bootstrap-vs-canonical semantics, config-version metadata — is defined by #11637 and documented in Configuration. A deployment's tenant config resolves through three tiers: the kb-config:<tenantId> graph node → the kb-config.yaml bootstrap → the local config.mjs defaults. This tiering now also covers tenantRepos (the pull-mode polling config), resolved via listConfiguredTenantRepos.
Tenant-repo checkpoint revalidation
Pull-mode deployments upgrading from a release that could persist a repository head after an error-bearing ingestion summary do not need to delete state or run an in-container recovery command. Unversioned checkpoint heads are treated as historically unknown and receive one automatic null-base replay.
Migration is gradual: the one-minute scheduler sweep admits no more than the
configured tenant-repo concurrency limit, while per-repo cadence, deterministic
jitter, failure backoff, and the normal concurrency semaphore remain active.
Failures preserve the old head and stay eligible for a later retry. Only a
clean Knowledge Base summary writes the current success-contract marker and
returns that repo to incremental ingestion. The deployment-state snapshot
exposes aggregate and hashed per-repo progress; scoped CLI --full replay
remains an optional acceleration override.
Breaking-change inventory
There are no breaking changes for an existing single-repo deployment. The substrate is additive end-to-end. The only "migration" a single-repo operator performs is git pull — the defaults handle the rest.
For a deployment that was manually patching the hardcoded source array or per-source paths (a pre-substrate fork): un-fork. Move the customization into aiConfig.customSources / aiConfig.sourcePaths. The registry + config substrate exists precisely to retire those forks.
Deprecation timeline
No deprecations in Phase 0/1. The legacy single-source-string chunk metadata is superseded by the path-identity tuple, but tenantId: 'neo-shared' / repoSlug: 'neo' is the deterministic default — no migration window, no dual-read shim. A KB re-sync (npm run ai:sync-kb) re-stamps all chunks under the tuple on the next run.
Related
- Overview — the contract split + default-source inheritance.
- Security — tenant-isolation invariants + spoof-rejection policy choice.
- Configuration — tenant-config persistence (#11637).