A source that silently under-produces makes kbSync delete its content

Context

Found while chasing a corpus loss that turned out to be something else entirely. This is not that incident — the loss of 2026-08-05 is attributed to a Chroma restart at 19:03:53.994Z and belongs to #16549. This ticket is the latent hazard the investigation surfaced on the way past, filed on its own terms and deliberately not citing that incident as evidence.

Narrowed twice before filing, both times by @neo-opus-vega, and the narrowing is most of the value: my first two framings were wrong in the safe direction — I described the hazard as bigger than it is.

The Problem

kbSync resolves to the delete-upfront stale strategy on this deployment, and the stale set is a set difference against a freshly generated file whose completeness is never checked.

ai/services/knowledge-base/VectorService.mjs:1073:

const idsToDelete = resolvedStaleStrategy === STALE_STRATEGY_SKIP
    ? []
    : existingIdsArray.filter(id => !allIds.has(id));

allIds is every chunk id in the JSONL that createKnowledgeBase just wrote. existingIds is what the collection holds. Anything in the collection and absent from that file is deleted. That is correct behaviour when the file is complete — it is exactly how genuinely-removed content leaves the corpus.

The hazard is that nothing establishes the file is complete before the difference is applied. createKnowledgeBase (DatabaseService.mjs:568) sums await source.extract(...) across SourceRegistry.getSources() and returns the total. A source that throws aborts the sync — fine. A source that silently yields fewer chunks does not: the file is smaller, the difference is larger, and every chunk that source used to contribute is deleted as stale.

Per-file failures inside a source are warnings, not errors. Observed on the live plane during an otherwise-healthy sync:

19:06:17.987Z [WARN] Failed to parse test file test/playwright/util/flickerCensus.mjs: Unexpected character '!' (1:1)

One file, one warning, sync continues. The same shape at directory scale — a mount that is absent, a path that moved, a source whose glob stops matching — produces a smaller file with no error and a proportional deletion.

What this is NOT, since I claimed both and both were wrong

• Not a wipe. It is a set difference. With a complete file the stale set is small.
• Not an oscillation. I claimed the corpus "returns to zero every 30 minutes" because the 30-minute cadence is far shorter than a full rebuild. Wrong: after a partial ingest every id in the collection came from the same JSONL, so allIds contains them all, idsToDelete is empty, and the next run resumes upserting chunksToProcess. It converges. The cadence-vs-runtime mismatch is real and harmless.

Recording both corrections because the overstated version is more alarming and would misdirect whoever picks this up.

The Architectural Reality

• ai/services/knowledge-base/VectorService.mjs:1073 — the difference. Correct given a trustworthy input.
• ai/services/knowledge-base/VectorService.mjs:273-281 — resolveStaleStrategy: absent strategy plus deleteStale = true resolves to delete-upfront.
• ai/scripts/maintenance/syncKnowledgeBase.mjs:86 — process.env.NEO_KB_STALE_STRATEGY || undefined; the variable is empty on the container plane (verified), so the scheduled lane takes the destructive default.
• ai/services/knowledge-base/DatabaseService.mjs:568 — createKnowledgeBase returns a chunk total that nothing compares against anything.
• shadow-swap already exists (VectorService:724-828) and is the non-destructive form: build a shadow, swap atomically, canonical stays populated throughout, resume-from-shadow on lease yield. It is a STALE_STRATEGIES member and is not the default for the scheduled lane.

The Fix

Direction, not prescription — two independent halves, and the first is worth landing alone.

1. Refuse the difference when the input is not credible. Compare the freshly-generated chunk total against the collection's current size and refuse to apply a stale set that would delete more than a configured proportion. A corpus does not lose a large fraction of itself in one sync without something being wrong upstream; that is a fail-closed check, not a heuristic.
2. A source that produces zero where it previously produced many should fail, not warn. Per-source counts are already computed in createKnowledgeBase; nothing retains or compares them across runs.
3. Consider shadow-swap as the default for the scheduled lane. A manual CLI resync can reasonably be destructive; a periodic unattended one arguably should not be. Listed third because it is the widest change and the first two make it optional rather than necessary.

Contract Ledger Matrix

Decision Record impact

none for items 1 and 2. Item 3 would change the scheduled lane's durability posture and should say so explicitly if it lands.

Acceptance Criteria

• A JSONL missing a whole source's output causes the sync to refuse, with the collection unchanged — proven by a spec that fails against today's code.
• A legitimately shrinking corpus (content genuinely removed, within the bound) still deletes, so the guard is not a blanket refusal — asserted with a positive control.
• A source that produced N chunks and now produces 0 fails the sync rather than warning.
• resolveStaleStrategy's existing behaviour and specs are untouched.
• The bound is a config leaf, not a literal, and its default is stated with the reasoning for the value.

Out of Scope

• #16549 — the 2026-08-05 corpus loss. Attributed to a Chroma restart at 19:03:53.994Z, which preceded kbSync starting at 19:06:10.068Z by 2m16s. This ticket is not that cause and must not absorb it.
• #16561 — the heavy-maintenance lease.
• #16557 — tenant-mirror read latency. Different lane, different producer.
• Why Chroma restarted.

Avoided Traps

• Describing the hazard as a wipe. It is a set difference, and the difference is empty in the common partial-ingest case. Stated because my own first two framings were louder than the defect.
• Welding a real finding to a coincident symptom. delete-upfront is genuinely the resolved default; that fact establishes nothing about whether it fired during any particular loss. Two events coinciding is not an order.
• Treating the 30-minute-vs-12.5-hour cadence mismatch as the defect. It converges. It is untidy, not dangerous.

Related

#16549 (the incident this is explicitly not) · #16563 (KB export reports success on zero rows — the same "an empty result is reported as fine" class on the export side) · #16554 / PR #16556 (gave this lane an owner, which is why its steady state is observable at all).

Live latest-open sweep: checked the latest 12 open issues by creation at 2026-08-05T19:26Z — #16563 is adjacent (export-side emptiness) but distinct from the ingest-side stale difference; no equivalent found.

Origin Session ID: eeacb603-97f1-4241-9b2f-3a542cab6d2c

Retrieval Hint: query_raw_memories("kbSync delete-upfront stale set difference allIds truncated jsonl source under-produces shadow-swap not default")