A parser-identity change orphans rows no reconciliation signal sees

Context

On a cloud deployment, rows carrying a superseded parser generation for a tenant repo remained in the neo-knowledge-base collection and continued to rank in query_documents results hours after that tenant's config had advanced its declared parserVersion. The rows were stamped tenantConfigVersion: 0; the advanced parser version was 1.1.0. Every post-bump tenant sweep reported deleted=0.

Separating observation from inference:

• Observed — row metadata carrying a parserVersion older than the declared one alongside tenantConfigVersion: 0; deleted=0 on each sweep; superseded rows returned in ranked query output above first-party content.
• Read from the code (cited below) — why no reconciliation signal classifies them as orphans.

This ticket's first draft asserted that no reclaim mechanism existed. That was false, and the falsification is the useful part. Phase 4B (#11640) shipped kbReconciliationEngine plus the KbReconciliationService daemon, and Phase 4C (#11641) shipped garbage collection; #11641's intake explicitly routed "config-orphan detection" to #11640 rather than dropping it. The mechanism is present and correct for the signals it has. What is missing is narrower: the parser identity is not one of those signals.

The Problem

A chunk id is identity-addressed over the parser generation — hashInputs is ['kind', 'name', 'content', 'sourcePath', 'parserId', 'parserVersion'] — so advancing parserVersion changes every id for the repo. The new generation adds rows; it never overwrites its predecessor. That is correct behaviour, and it is what makes a generation change detectable at all. The defect is that nothing detects it.

kbReconciliationEngine classifies orphans by exactly two signals, and a parser-identity change moves neither:

Signal 1 — config invalidation (diffTenantChunks). A row is a config-stale orphan when metadata.tenantConfigVersion is strictly below the tenant's current getTenantConfig().version, becoming actionable at versionGap >= 2 (one epoch of grace, DEFAULT_ORPHAN_VERSION_GAP). The stamp is written as tenantConfigVersion: tenantContext.configVersion ?? 0, sourced from the tenant config version. Nothing ties it to {parserId, parserVersion}. A repo can advance its declared parser version repeatedly while every row — old generation and new — stays stamped at the same tenantConfigVersion, so versionGap is 0 and no row is ever stale.

Signal 2 — claimed-state manifest (diffTenantManifest, #11711). A row is a manifest orphan when its metadata.sourcePath no longer appears in the persisted kb-manifest:<tenantId> for its repoSlug, and only when metadata.ingestedAt is at or before the manifest's updatedAt. This signal is deliberately decoupled from the config version so routine push manifests do not bump config staleness — it is push-path substrate, and it says nothing about a row whose path is still present but whose parser generation is superseded.

The consequence splits, and only one half self-heals. For a sourcePath the current parser version still emits, the superseded row is eventually joined by its replacement: storage is wasted and both generations answer queries, but no content is lost. For a sourcePath the current version no longer emits at all — the case that arises when a directory is added to a parser's vendor-exclusion list — no re-emission can ever supersede that row. Nothing will write that id again, and no signal classifies it. It is permanent.

Those permanent rows stay fully queryable and rank against first-party content, so retrieval precision degrades by exactly the share of corpus that was excluded — the opposite of what excluding it was for. An agent asking about first-party code is served vendored library internals from a parser generation that no longer exists.

Second-order: the collection becomes a mixture of generations, so its count cannot serve as ingestion-progress evidence. Any rate or completion estimate derived from it conflates new ingest with unreclaimed residue.

A second, independent gate is worth stating so the fix is not mistaken for sufficient. aiConfig.knowledgeBase.reconciliationEnabled defaults to false and KbReconciliationService.start() is a documented no-op when it is unset. On a deployment that has not opted in, no reclaim runs regardless of signals. That default is deliberate and is not what this ticket changes — but a reader who fixes only the signal will still observe no reclaim, and a reader who only flips the flag will still observe no reclaim for parser-identity orphans. Both are required, and they have different owners.

The Architectural Reality

• ai/services/knowledge-base/helpers/kbReconciliationEngine.mjs — the pure classifier. Its module JSDoc states the V1 signal is tenantConfigVersion and the V1.x signal is manifest membership; diffTenantChunks and diffTenantManifest are the two entry points, and neither reads parserId / parserVersion.
• ai/services/knowledge-base/VectorService.mjs:349 — tenantConfigVersion: tenantContext.configVersion ?? 0, the stamp. :163 lists TENANT_GUARDED_FIELDS, which includes tenantConfigVersion and ingestedAt but no parser-identity field.
• ai/services/knowledge-base/IngestionService.mjs:2287 / :2315 and ai/services/knowledge-base/source/RawRepoSource.mjs:147 — the hashInputs tuple that puts parserId / parserVersion into the id.
• ai/configBase.mjs:2395 — reconciliationEnabled: false; :2411 documents the version-gap threshold.
• ai/daemons/kb-reconciliation/KbReconciliationService.mjs:116 — the early return when reconciliation is disabled.
• ai/services/knowledge-base/VectorService.mjs:2396 — the incremental reclaim is a set-difference over the incoming corpus (existingIdsArray.filter(id => !allIds.has(id))), skipped entirely under STALE_STRATEGY_SKIP. It is sound only over a complete pass, so it is not an alternative route for a lane that embeds in bounded slices. :2470 is the id-based collection.delete; there is no metadata-predicate variant. :2331 records the prior incident where an unscoped idsToDelete treated other tenants' rows as stale — ownedScope is the read-filter that shipped in response and the precedent any new scoping must honour.
• ai/services/knowledge-base/helpers/kbGarbageCollectionEngine.mjs:22 — already mirrors "#11640's missing-tenantConfigVersion skip", so the sibling engine's skip semantics are the precedent for how an unstamped row is handled.
• ADR 0017 §2 — realm/tenant separation is enforced by "collection names + per-chunk metadata … write-stamping + read-filter model", never by directory or daemon split; §5 records that all tenants share one neo-knowledge-base collection, metadata-filtered. A metadata-keyed orphan signal is therefore in-idiom.

The Fix

Add parser identity as a reconciliation signal, reconciled against declared config rather than triggered by a version-change event.

The distinction is load-bearing. A design that fires when parserVersion changes cannot clear an already-orphaned set without another bump, and a bump re-materializes the entire corpus — on a deployment partway through its initial embedding that costs far more than the orphans do. Classifying a row by comparing its stamped {parserId, parserVersion} against the repo's currently declared pair makes an existing orphan set self-healing on the next ordinary reconciliation tick, with no bump.

Shape it as a third pure classifier in kbReconciliationEngine, alongside its two siblings and with the same no-I/O contract:

1. Unyielded-path orphans — no replacement pending, immediately actionable. A row whose sourcePath the currently declared parser yields nothing for has no replacement coming, so reclaiming it opens no retrieval hole. This is the tier that clears a vendor-exclusion change and the tier that needs no bump. Membership must come from an authority that is complete now: the tenant sweep materializes an envelope enumerating the whole tree on every pass while embedding proceeds in bounded slices, so path membership is knowable long before the repo finishes.

2. Superseded-generation rows — replacement-gated, per path. For a sourcePath still yielded, classify rows carrying a non-declared {parserId, parserVersion} as actionable only once that path's replacement row exists. diffTenantManifest's existing ingestedAt <= manifest.updatedAt guard is the precedent for this shape of gate: authority is scoped to what the signal can actually see.

Neither tier deletes ahead of its replacement. #16577 measured a materialization that reported success while leaving no durable proof, which makes the delete-before-embed window concrete rather than theoretical, and #16611 dispositioned staleStrategy's destructive branch as deliberately not remotely selectable. This ticket adds no remote selector for a destructive branch and does not change the reconciliationEnabled default.

Home: ai/services/knowledge-base/helpers/, extending kbReconciliationEngine.mjs beside kbGarbageCollectionEngine.mjs and kbAlertRuleEngine.mjs — the established pure-helper trio.

Contract Ledger Matrix

Ledger amended 2026-08-19. The original matrix omitted the telemetry surface entirely — formatReconciliationDetail's detail payload gains a key when this signal lands, and no row named it. Caught while reviewing PR #17395, which ships that key. The omission was mine; the row above is the repair.

Decision Record impact

aligned-with ADR 0017 — metadata-scoped, collection-shared tenant isolation is the mandated mechanism, and this uses it for orphan classification rather than adding a directory or daemon split. depends-on ADR 0014 for the identity-tuple write-stamping model that makes row identity readable, and aligned-with ADR 0013 for the telemetry surface a reclaim count reports through. No ADR is amended, superseded, or challenged.

Acceptance Criteria

• Red-proof: against the current tree, a probe asserting "no row within a repo's ownedScope carries a {parserId, parserVersion} other than the declared pair" must FAIL, reproducing the orphan state. If it passes on dev, it is not exercising this defect.
• A repo whose declared parser version advanced while a directory became vendor-excluded has those rows classified actionable on the next reconciliation pass, with no further parserVersion bump.
• Control: a repo whose declared {parserId, parserVersion} is unchanged classifies nothing. Present and labelled, because without it a green suite is equally consistent with the classifier firing indiscriminately.
• tenantConfigVersion independence: the new classifier fires on a fixture where tenantConfigVersion is identical across both generations, proving it does not depend on the signal that already exists and already fails here.
• Tenant containment: with two tenants resident in the one shared collection, classification for tenant A yields zero rows carrying tenant B's tenantId. Asserted by per-tenantId counts, not by the classifier's own report.
• No-hole: for a sourcePath the declared parser still yields, no row is classified actionable until that path's replacement exists. Asserted at partial embedding progress, observing the superseded row survive.
• A row missing a parser-identity stamp is skipped rather than classified, matching kbGarbageCollectionEngine's missing-stamp precedent.
• The reconciliation telemetry distinguishes parser-identity orphans from config-stale and manifest orphans, so one cannot be read as another.
• No new MCP-exposed selector for a destructive branch, and no change to the reconciliationEnabled default.

Out of Scope

• Flipping reconciliationEnabled. The opt-in default is deliberate; enabling it on any given deployment is a deployment decision, not this ticket. Named in The Problem only so the fix is not mistaken for sufficient on its own.
• The vector-generation election. A plane reporting vectorGeneration: status 'missing' never declared a baseline and runs in legacy mode for the promote fence. declareBaselineVectorGeneration only initializes a plane without a record — it does not retroactively reclaim, so it is not an alternative route.
• staleStrategy's delete-upfront branch and its NEO_KB_STALE_STRATEGY operator surface.
• Changing hashInputs. Including the parser identity in the id is correct; it is what makes a generation change detectable.
• Heavy-maintenance lease fairness, embedding-lane throughput, and chunk sizing. Adjacent, real, separately ticketable.

Avoided Traps

• Asserting the mechanism was absent. This ticket's first draft claimed no reclaim path existed and prescribed building one. Reading kbReconciliationEngine killed that: Phase 4B/4C shipped both a reconciliation daemon and a GC engine, and #11641's intake deliberately routed config-orphan detection to #11640 rather than dropping it. Recorded because the wrong version of this ticket is the one a successor will re-derive.
• Triggering on the bump. Cannot clear an existing orphan set without another bump, which re-materializes the whole corpus. Reconciling against declared config is what makes the current state self-healing, and it is why this is shaped as classification rather than as an event handler.
• Reusing the incremental set-difference. Under bounded-slice progress it would delete every row not in the current slice. The path-membership predicate is available precisely because the envelope is complete while embedding is not.
• Deleting before the replacement lands. Split into two tiers for this reason; #16577 makes the window concrete.
• Reading collection count as ingestion progress. A mixed-generation collection makes count-derived rates and ETAs unsound. Recorded so a successor does not re-derive throughput from a number that conflates two generations.
• Treating this as the ownedScope fix's leftover. That fix correctly stopped cross-tenant over-deletion; it did not and could not make a slice-bounded set-difference safe.

Related

• #11640 — Phase 4B reconciliation daemon; owns the domain and shipped the two signals this extends.
• #11641 — Phase 4C stale-chunk GC; its intake routed config-orphan detection to #11640.
• #11711 — the claimed-state manifest signal (diffTenantManifest).
• #11712 — server-stamped ingestedAt, the stamp the manifest gate depends on.
• #16577 — the materialization that reported success while leaving no durable proof; the delete-before-embed anchor.
• #16611 — dispositioned staleStrategy's destructive branch as not remotely selectable.
• ADR 0017 / ADR 0014 / ADR 0013.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-19T17:08:53Z; no equivalent found. state:all keyword sweeps on parserVersion, stale chunks reclaim, superseded rows, excludePaths, and orphan chunks surfaced #11640 / #11641 / #16549 / #16584 — all read, all distinct, and the two that own this domain are cited above rather than duplicated. A2A in-flight claim sweep at the same timestamp: no [lane-claim] / [lane-intent] overlapping this scope.

Origin Session ID: 8cbd588b-be06-4a56-9997-1058f2a3a07b

Retrieval Hint: query_raw_memories("parser identity is not a reconciliation signal tenantConfigVersion unchanged across parser version bump vendor exclusion orphan rows kbReconciliationEngine")