A restored Knowledge Base collection is not durable: retrieval works, a restart discards it, and the next mutation persists the emptiness

Refs #16510 · #16512 · #16521 · second corpus loss, distinct mechanism from the first.

Context

The neo-knowledge-base collection has now been lost twice, by two different mechanisms.

Loss 1 (dockerization migration) was a recreation: a fresh empty collection took the canonical name under a new id (a9637b4c… → ab75f86b…) while ~61,206 documents stayed behind in the previous data root. Nothing deleted anything.

Loss 2 is this ticket. @neo-opus-grace restored 61,206 chunks at 2026-08-04T19:44Z and verified retrieval with varying scores. The collection is now empty with the same id — ab75f86b, still canonical. So the collection was not replaced; it lost its contents in place.

The measured timeline

Current state, live: neo-knowledge-base count 0, stable across three reads. Siblings through the same endpoint: neo-agent-memory 32,419 · neo-agent-sessions 1,941 · graph 1,529 · temporal 69. The siblings are the positive control — same instance, same mechanism, populated.

The Problem

The compaction pointer and the segment content disagree, and only for this collection:

The pointer claims everything up to 4423520 was folded into the segment. The segment holds nothing. Corroborating magnitude: 61,206 vectors at 4096 dims is on the order of 1 GB; the KB segment is 1.6 MB, and the whole chroma-data volume is 1.9 GB.

So the data was queryable at 19:44Z and never durably on disk — an in-memory index serving reads over an empty persisted state. The 20:03Z restart discarded that index; Chroma reloaded from a disk state holding nothing for this collection; the 21:10:52 single-document delete then forced an index rebuild from empty and persisted a 124-byte result.

The delete is not the cause. It is the first mutation that made the loss visible. A guarded mass-delete would have written 61,206 records or advanced the pointer far past 4423520.

What is established vs. what is not

Established: the timeline above, the pointer/segment disagreement, the id stability, and that restores go through the ordinary client path — restore.mjs:64 uses collection.get({ids}) and collection.add() via KB_DatabaseService.manageDatabaseBackup({action:'import'}), so this is not a queue bypass. The adds' absence from embeddings_queue is compaction pruning, which is steady state.

NOT established — and the ticket must not assert it:

• What caused the 20:03Z restart. The orchestrator's Aug-4 log returns 0 lines for 19:50–20:10 because a rebuild on 2026-08-05 recreated that container and destroyed its history. Control included: zero lines, so no attribution is claimed.
• Whether the restart is causal or coincident. It is mechanically sufficient and correctly ordered; that is not the same as proven.
• Whether this is Chroma-side or restore-side. A collection.add() that is acknowledged, queued, pruned by compaction, and never written to the segment is a durability-contract violation wherever it lives.

Ruled out, each by measurement rather than argument:

• Recreation, shadow-swap promotion, deleteCollection+recreate, defrag-resume — all four require an id change, and the id never moved. createSwapCollectionName mints ${name}-${phase}-${timestamp}-${uuid}, so any promotion yields a new collection.
• Autonomous defrag resuming a broken phase — defragChromaDB.mjs:1531 passes non-empty allowedPhases only for memory-core; the KB target passes [] and therefore refuses.
• A volume removal — neo-local-agent-os_chroma-data was created 2026-07-31T06:56:53 and never recreated, and four collections survive on it.
• Disk exhaustion — weakly supported at best: 26 GB free, oldest build-cache entries 2 months old, and no ENOSPC or write error logged by Chroma in the 20:00–22:00 window, with the log proven to cover Aug-4 (56 lines that day).

The discriminating experiment, and it is blocked

Create a throwaway collection, add() a batch, confirm the count reads back, restart Chroma, re-count.

• If the rows vanish with no mutation at all, durability is broken and the delete was never even the trigger.
• Two obstacles: compaction is asynchronous, so an immediate zero-segment reading proves nothing without waiting for or forcing a fold; and a Chroma restart takes the whole plane's vector store down, which is operator authority.
• It cannot be run via restore.mjs — that tool has no target-collection flag, so the obvious version of this experiment would restore 61,206 rows into the live canonical collection. Tracked separately, and that ticket unblocks this one.

Acceptance Criteria

• The experiment above is run and its outcome recorded, including the compaction-timing control that distinguishes "not yet folded" from "never persists".
• A restored collection's rows are verified durable across a restart before a restore is reported successful — a restore that cannot survive a restart has not restored anything.
• The pointer/segment disagreement is either impossible by construction or detected: something must refuse to advance a compaction pointer past rows the segment did not receive, or must report the mismatch.
• A spec asserts both directions — a durable batch survives, and a non-durable one is reported rather than silently accepted.
• The 20:03Z restart's cause is attributed, or recorded as unattributable with the reason (the log was destroyed by a mid-incident rebuild).

Out of Scope

• RESTORABLE's two roles — #16521 owns that. This ticket is about whether a restore lands, not about what the verdict promises.
• The Chroma persist-path mismatch (#16208) — the current topology contradicts it: persist_path: "/data" on a real device, and siblings persist correctly.
• Re-restoring the corpus. Deliberately held: if the hypothesis holds, restoring the same way reproduces the loss on the next restart or mutation. Both bundles are intact and that matters more than a fast recovery.
• The orchestrator OOM (~1 GB heap during tenant repo sync) — a separate live defect. The cadence-based link between it and this loss was proposed and then retracted: 21:11 is 87 minutes post-restore, not an overnight profile.

Avoided Traps

• Reading the collection listing as evidence of content. The id was stable, which I nearly published as "no recurrence." The listing shows identity; only the count shows content.
• Treating an expected absence as a finding. The adds' absence from embeddings_queue is compaction, not bypass — a conditional consequence built on that premise was retracted.
• Cleaning up before measuring. ~5.8 GB of stale images from an unrelated build remain on the host deliberately: disk state is evidence while this is open, and reclaiming it destroys the ability to measure what the pressure was.

Related

• #16521 — the verdict's semantics; adjacent, not this.
• #16510 — the per-collection restorability verdict that made loss 1 legible.
• #16512 — the health/render masking. Merged 2026-08-05T11:37:50Z but NOT in the running image, so this loss is currently silent on the plane.

Live latest-open sweep: 40 most recent open issues plus targeted searches for restore/durability/chroma-restart/persist; nearest neighbours are #16521, #16208, #15639, #15693 and none owns in-place loss of a restored collection.

Origin Session ID: 11695cce-9854-4be2-80c3-8ea4322298bf

Retrieval Hint: query_raw_memories("restored collection not durable chroma restart discards in-memory index compaction pointer advanced empty segment 124 byte pickle")