defragChromaDB orphan-cleanup deletes live HNSW segment dirs in the unified store

Context

The orchestrator-managed Chroma daemon now runs a single unified store (ADR 0003: chroma-topology-unified-only). One daemon on localhost:8000 persists to .neo-ai-data/chroma/knowledge-base/, holding all collections for both Knowledge Base (neo-knowledge-base) and Memory Core (neo-native-graph + per-session memory/session collections).

#12138 will have the orchestrator recycle that daemon (kill → restart → defrag) on a max-runtime cadence. The "defrag" step shells out to ai/scripts/maintenance/defragChromaDB.mjs (the "Nuke and Pave" tool; npm scripts ai:defrag-kb / ai:defrag-memory). Before the orchestrator can run that script automatically, the script must be safe for the unified store. Today it is not.

Problem (DATA LOSS)

defragChromaDB.mjs step 6 ("Cleanup (Physical)") physically deletes on-disk UUID directories that are not in its keep-set:

• Keep-set is built from recreated collection ids: newCollectionIds.push(newCollection.id) — ai/scripts/maintenance/defragChromaDB.mjs:434.
• Deletion scans DB_PATH and removes any UUID dir whose name is not in that set: if (!newCollectionIds.includes(entry.name)) { await fs.remove(orphanPath) } — defragChromaDB.mjs:471-480.

But the on-disk UUID directories in a Chroma persist dir are named by segment id, not collection id. Each collection has VECTOR + METADATA segments with their own UUIDs; only VECTOR (HNSW) segments get a physical directory. Collection ids and segment ids are disjoint UUID spaces, so newCollectionIds.includes(entry.name) can never match a real segment dir.

Empirical ground truth (current unified store, point-in-time snapshot 2026-05-28):

• collections = 1138, segments = 2276, on-disk UUID dirs = 307.
• Of those 307 dirs: 17 are live VECTOR segments, 0 are METADATA, 290 are true orphans (absent from the segments table).
• The keep-set (collection ids) matches zero of the 17 live segment dirs.

Therefore step 6 currently deletes all 307 dirs — including the 17 live VECTOR segment dirs for every collection in the store (KB and MC). The running daemon survives on in-memory/sqlite state, so the damage is latent until the next daemon restart — which is exactly what #12138's recycle (kill → restart) performs. After restart, every collection's HNSW index dir is gone → vector queries return nothing across both subsystems.

This is target-agnostic: ai:defrag-kb (DB_PATH = cfg.path = .neo-ai-data/chroma/knowledge-base) operates directly on the live unified dir and wipes MC's live segments along with KB's.

Architectural Reality

• Unified topology: ADR 0003 (learn/agentos/decisions/0003-chroma-topology-unified-only.md). One daemon, one persist dir, all collections.
• Daemon launch path: ai/daemons/orchestrator/TaskDefinitions.mjs:62 → chroma run --path .neo-ai-data/chroma/knowledge-base.
• KB defrag target path: defragChromaDB.mjs TARGETS knowledge-base.adapt → path: cfg.path (= .neo-ai-data/chroma/knowledge-base, ai/mcp/server/knowledge-base/config.template.mjs:132). This is the live unified dir.
• Segment-vs-collection id authority: Chroma's segments table in <DB_PATH>/chroma.sqlite3 maps segment.id → scope → collection.id. On-disk VECTOR dirs are named by segment.id. Verified by cross-referencing on-disk dir names against segments.id / collections.id.
• The script already reads the sqlite via the sqlite3 CLI (vacuumSqlite, defragChromaDB.mjs:219-230, execSync('sqlite3 ... "VACUUM;"')), so the authoritative segment registry is reachable with the same pattern — no new dependency.

The Fix (one prescription)

Replace the collection-id keep-set with the authoritative live-segment-id set:

1. After the delete+recreate phase, read all live segment ids from the registry: SELECT id FROM segments in <DB_PATH>/chroma.sqlite3 (same sqlite3-CLI access pattern as vacuumSqlite).
2. In step 6, delete an on-disk UUID dir iff its name is not in the live-segment-id set.

This removes the true orphans (defrag's actual purpose, per prior art #9579) while preserving every live VECTOR segment dir for every collection — making the tool target-agnostic and unified-store-safe.

Extract testable seams following the existing export pattern (cleanOldBackups, resolveDefragSnapshotRetention):

• export async function resolveLiveSegmentIds({dbPath, execFn}) → Set<string>
• export async function cleanOrphanedSegmentDirs({dbPath, liveSegmentIds, fsModule = fs, log})

Accept and document the coupling to Chroma's segments schema: the script is already a low-level physical-maintenance tool that deletes Chroma's segment dirs and vacuums its sqlite, so reading the segments table is at the same abstraction level. Sunset note: if a future chromadb client exposes segment ids, prefer that over direct sqlite reads.

Contract Ledger

Acceptance Criteria

1. Step-6 keep-set is sourced from the live segments registry (segment ids), not recreated collection ids.
2. Running ai:defrag-kb against a unified store containing multiple collections preserves all other collections' live VECTOR segment dirs (no MC data loss).
3. True orphans (UUID dirs absent from the segments table) are still removed (defrag retains its compaction value).
4. Orphan-cleanup logic is extracted into exported, unit-testable helpers (resolveLiveSegmentIds, cleanOrphanedSegmentDirs) with injectable fs/exec seams, matching the existing cleanOldBackups pattern.
5. A unit test includes a negative-mutation assertion: a temp dir seeded with this-target live dir, a sibling-collection live dir, a true-orphan dir, and a non-UUID dir → asserts orphan removed and both live dirs + non-UUID dir preserved. Run via npm run test-unit.

Out of Scope

• The Chroma daemon recycle chain itself (#12138) — this ticket only makes the defrag step safe so the recycle can call it.
• ai:defrag-memory's DB_PATH resolving to the stale legacy .neo-ai-data/chroma/memory-core dir (separate config-correctness defect: MC's engines.chroma.dataDir, ai/mcp/server/memory-core/config.template.mjs:242, predates ADR 0003 unification and no longer points at the live store). Triaged separately.
• Test-collection pollution: ~1100 test-* collections present in the production unified store (tests writing to localhost:8000 rather than an isolated store). Separate test-isolation defect. Triaged separately.

Related

• #12138 — Orchestrator recycles Chroma after a max runtime (blocked by this).
• #12139 — KB + MC external-only Chroma; orchestrator as single source of truth.
• PR #12137 — Orchestrator guarantees a single Chroma daemon (merged).
• Prior art: #9579 (defrag ghost-entry extraction), #8490 (defrag hardening/refactor).
• ADR 0003 — Chroma Topology: Unified Only.

Decision Record

• aligned-with ADR 0003 (chroma-topology-unified-only): the unified store is precisely why a target-scoped / collection-id keep-set is unsafe; the fix re-derives the keep-set from the instance-wide segment registry.

Handoff Retrieval Hints

• Keywords: defrag, Chroma, segment id vs collection id, HNSW segment dir, unified store, step-6 orphan cleanup, segments table, data loss on restart.
• Durable anchors: #12138, PR #12137, ADR 0003, defragChromaDB.mjs:434 / :471 / :219.