Harden Chroma defrag against interrupted canonical rebuild

Context

During the 2026-06-18 Chroma integrity follow-up, the malformed FTS5 index investigation surfaced a separate safety issue in ai/scripts/maintenance/defragChromaDB.mjs: the script buffers collection data, deletes canonical collections, recreates them, and reloads the buffered rows. A normal run can complete successfully, but a hard kill or process crash in the delete/recreate/reload window can leave a canonical collection missing or only partially restored until an operator rebuilds/restores it.

This is not the same bug as #13467. #13467 tracks the malformed Chroma internal FTS5 index and explicitly keeps defrag redesign out of scope. #13469 adds only the copy-first diagnostic script.

Live latest-open sweep: checked the latest 20 open GitHub issues at 2026-06-18T13:54:02Z; adjacent tickets #13466, #13467, and #13469 exist, but no equivalent defrag-interruption hardening ticket was found.

A2A in-flight claim sweep: checked latest 30 messages via list_messages({status:'all', box:'all', limit:30}); only my related #13466/#13467/#13469 Chroma lane messages were present, with no competing [lane-claim] or [lane-intent] for defrag interruption hardening.

Local exact sweep: searched resources/content/issues, resources/content/discussions, ai, learn, and .agents for defrag hard kill, delete/recreate, canonical collection absent, shadow-swap defrag, and defragChromaDB; found existing defrag references and adjacent architecture docs, but no ticket for this failure mode.

KB semantic sweep: attempted ask_knowledge_base for the defrag/shadow-swap pattern, but the local KB MCP was stale/unavailable (Failed to access collections: The requested resource could not be found, runtimeFreshness.openApiDigest=true). I did not rely on KB output for this ticket.

Release classification: post-release operational hardening by default; boardless unless the operator promotes Chroma maintenance safety to release-blocking.

The Problem

defragChromaDB.mjs currently implements a "Nuke and Pave" collection rewrite:

• It creates a physical pre-nuke snapshot of the unified Chroma persist directory.
• It extracts each target collection into an in-memory buffer.
• It deletes the canonical collection(s) through the raw Chroma client.
• It recreates each canonical collection and reloads the buffered data.
• It cleans orphaned segment dirs and runs SQLite VACUUM.

The hazardous window is between canonical delete and successful full reload. If the process is killed there, the snapshot exists under dist/chromadb-backups/<target>/, but the live Chroma daemon can remain without the canonical collection or with an incomplete replacement. Nothing writes a durable phase marker, nothing resumes or rolls back automatically, and a later healthcheck may only report that the collection is missing.

The risk is sharper in local Neo than it first appears because the orchestrator can automatically run the KB defrag after recycling Chroma. AiConfig.orchestrator.chroma.maxRuntimeMs defaults to a nonzero value, Orchestrator.poll() sets _chromaDefragPending after Chroma recycle, and the chromaDefrag task invokes defragChromaDB.mjs --target knowledge-base after the fresh daemon is connection-ready.

The Architectural Reality

Relevant anchors:

• ai/scripts/maintenance/defragChromaDB.mjs lines 361-373 create a defrag-private physical snapshot, not the canonical JSONL backup.
• ai/scripts/maintenance/defragChromaDB.mjs lines 472-502 delete canonical collections, then recreate them under the same names.
• ai/scripts/maintenance/defragChromaDB.mjs lines 535-549 clean orphaned segment dirs and run VACUUM after the rewrite.
• learn/agentos/decisions/0017-chroma-single-flat-unified-store.md defines one flat chroma/unified store shared by KB and MC; recovery must be collection-scoped, not folder deletion.
• learn/agentos/tooling/RestorationRunbook.md states KB is a rebuildable cache but MC is an irreplaceable store; both live inside the same unified Chroma store.
• ai/services/knowledge-base/VectorService.mjs already has a safer KB full-corpus pattern: build a shadow collection, rename live to parking, rename shadow to canonical, and keep the parked live collection as a rollback artifact.
• ai/services/knowledge-base/ChromaManager.mjs blocks cold-cache empty canonical recreation while active shadow/parking collections exist.
• ai/services/shared/vector/chromaClientPrimitives.mjs has a guarded delete wrapper for canonical collection deletes, but defragChromaDB.mjs currently uses the raw client.deleteCollection() path directly.
• ai/daemons/orchestrator/taskDefinitions.mjs defines the one-shot chromaDefrag task as defragChromaDB.mjs --target knowledge-base.

This means the correct fix is not a folder-level restore or a generic SQLite repair. It is a collection-scoped interruption-safe rewrite strategy, with different treatment for KB-as-cache and MC-as-store.

The Fix

Harden defrag so canonical collections are never absent for the duration of a full extraction/reload.

Expected shape:

• For --target knowledge-base, replace delete/recreate with a shadow/parking promotion model: load extracted data into a process-unique shadow collection, validate count/sample reads, then perform the bounded rename promote (live -> parking, shadow -> canonical) already used by KB shadow-swap semantics.
• For --target memory-core, either implement a safe multi-collection shadow/parking set with all collection counts validated before promotion, or fail closed behind an explicit operator-gated recovery mode until such a collection-set promotion exists. Do not delete neo-agent-memory, neo-agent-sessions, or neo-native-graph before replacements are fully loaded and validated.
• Add durable phase metadata for any non-atomic promotion window: target, source counts, shadow names, parking names, snapshot path, phase, and timestamp. A subsequent invocation should detect an incomplete prior run and either refuse with exact recovery instructions or continue/rollback only when the state is unambiguous.
• Ensure orchestrator-triggered chromaDefrag uses the hardened path and does not auto-launch the old delete/recreate behavior.
• Keep #13467's FTS5 repair separate; this ticket does not assume defrag fixes SQLite embedding_fulltext_search corruption.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR 0017. The fix must preserve the single flat unified Chroma store and strengthen collection-scoped recovery inside it.

Acceptance Criteria

• defragChromaDB.mjs --target knowledge-base no longer deletes the canonical KB collection before a replacement collection is fully loaded and validated.
• The KB defrag path uses or mirrors the existing shadow/parking promotion semantics rather than raw canonical delete/recreate.
• The Memory Core defrag path either becomes interruption-safe across all MC collections or fails closed with explicit operator recovery guidance.
• Defrag writes enough durable phase metadata to diagnose and recover from an interrupted promotion window.
• Orchestrator-triggered chromaDefrag cannot auto-run the old unsafe delete/recreate behavior.
• Tests cover the hard-kill windows at minimum by simulating failure before promotion, between live parking and shadow promotion, and during reload/validation.
• Documentation or script output distinguishes this defrag interruption hardening from the malformed FTS5 repair tracked in #13467.

Out of Scope

• Repairing malformed inverted index for FTS5 table main.embedding_fulltext_search; that remains #13467.
• Adding more healthcheck work to KB/MC hot paths.
• Folder-level deletion or restore of .neo-ai-data/chroma/unified; the unified store also contains MC data.
• Reworking Chroma's internal SQLite schema.

Avoided Traps

• Do not treat the defrag-private physical snapshot as a canonical backup substitute; backup.mjs and restore.mjs own portable JSONL backup/restore.
• Do not fix KB only and assume the MC target is safe; MC is the irreplaceable store and has stricter loss semantics.
• Do not run REINDEX, VACUUM, or manual SQLite repair as part of this ticket's proof. That belongs to #13467 after copy-first validation.
• Do not use raw client.deleteCollection() against canonical production collections unless the new design proves the failure windows are safe.

Related

Related: #13466 Related: #13467 Related: #13469 Related: PR #13470

Origin Session ID

Origin Session ID: 4ce60429-2986-4543-be2d-741957c75b6c

Handoff Retrieval Hints

• query_raw_memories: defragChromaDB nuke pave delete recreate collection hard kill restore window
• query_raw_memories: shadow swap knowledge base canonical collection live parked rollback Chroma rebuild
• query_raw_memories: Chroma FTS5 malformed inverted index quick_check repair defrag Error finding id