Repair Memory Core Chroma stored-embedding export

Context

During the #13467 Chroma integrity investigation, the on-demand integrity diagnostic separated two different facts:

• SQLite pragma quick_check / pragma integrity_check on a copied Chroma DB reported malformed inverted index for FTS5 table main.embedding_fulltext_search.
• Chroma API reads showed neo-knowledge-base healthy through count, get, stored-embedding fetch, and query-by-existing-embedding, but all Memory Core production collections failed stored-embedding materialization.

The escalated command was:

npm run ai:check-chroma-integrity -- --json --keep-snapshot

Observed production collection results:

neo-knowledge-base count=30466, get embedding by id=ok, query by existing embedding=ok
neo-agent-memory count=18843, get metadata/document=ok, get embedding by id=Error finding id
neo-agent-sessions count=1072, get metadata/document=ok, get embedding by id=Error finding id
neo-native-graph count=716, get metadata/document=ok, get embedding by id=Error finding id

The query path is not currently dead: the existing on-demand query canary passes:

node ai/scripts/maintenance/probeCollectionQueryHealth.mjs

returned:

{"status":"healthy","collections":{"memory":{"status":"healthy","count":18843},"summary":{"status":"healthy","count":1072}}}

So this is not a duplicate of closed #12450's final close premise. Query recall can be healthy while stored-embedding export is broken.

Live latest-open sweep: checked latest 20 open issues at 2026-06-19T01:06:10Z; no equivalent ticket found. Closest open item is broad REM epic #12065, not this Chroma stored-embedding export failure. A2A in-flight claim sweep: checked latest 30 all-status messages at the same time; no overlapping peer [lane-claim] / [lane-intent] found.

Release classification: post-release / operational hardening. Current recall canaries are healthy, but backup/restore and repair safety are materially degraded until this is resolved.

The Problem

Memory Core Chroma collections can still count records, fetch IDs, fetch metadata/documents, and pass a query canary, but cannot materialize stored embeddings via collection.get({include:['embeddings']}) for sampled production IDs.

That distinction matters because several maintenance paths depend on stored embeddings, not just queryability:

• backup/export paths need to preserve embeddings in JSONL records;
• defrag/rebuild paths need to extract embeddings before rewriting collections;
• repair diagnostics need to distinguish healthy query HNSW from broken stored-embedding retrieval.

A healthcheck or query canary that says "healthy" is therefore not sufficient evidence that Memory Core is safely exportable or repairable.

The Architectural Reality

Relevant surfaces:

• ai/scripts/maintenance/checkChromaIntegrity.mjs already exercises get embedding by id and exposed this failure.
• ai/scripts/maintenance/probeCollectionQueryHealth.mjs only verifies query-path health for Memory Core collections; it correctly passed in this state.
• ai/services/memory-core/DatabaseService.mjs export uses collection.get({include: ["documents", "embeddings", "metadatas"]}), then falls back item-by-item and skips corrupted vector IDs.
• ai/scripts/maintenance/defragChromaDB.mjs extracts {ids, embeddings, metadatas, documents} before any rewrite and is therefore affected by stored-embedding fetch failures. Memory Core defrag is currently disabled until a safe multi-collection path exists, which is the right default.
• ai/scripts/maintenance/backup.mjs explicitly states the canonical backup path does not preserve Chroma's physical HNSW indexes; restore re-ingests through service boundaries. That makes stored-embedding export correctness load-bearing.

The Fix

Investigation-first, repair-second:

1. Reproduce the Memory Core stored-embedding failure with a bounded probe that samples multiple IDs per production MC collection and reports counts, successes, and failures.
2. Determine whether DatabaseService.exportDatabase() currently produces partial/empty backups for affected MC collections, or whether it can rescue enough records to be useful.
3. Identify the safe repair/rebuild path for Memory Core collections when query works but stored embedding materialization fails. Candidate shapes include re-embedding from graph/Chroma documents, service-boundary export/import without preserved embeddings, or a purpose-built MC collection rebuild. Do not assume FTS5 rebuild fixes this class; the FTS5 copy-rebuild only fixes SQLite integrity.
4. Add or extend an on-demand diagnostic so operators can verify exportability separately from queryability.
5. Document the pre-repair backup/stop-daemon sequence before any live mutation.

Contract Ledger Matrix

Decision Record impact

none. This is operational integrity for Chroma-backed Memory Core collections and maintenance tooling, aligned with ADR 0017's unified Chroma topology.

Acceptance Criteria

• A bounded diagnostic reproduces stored-embedding materialization failure for neo-agent-memory, neo-agent-sessions, and neo-native-graph without mutating live data.
• The diagnostic reports query-path health and stored-embedding exportability as separate axes.
• Memory Core backup/export behavior is characterized: complete, partial-with-explicit-warning, or unsafe/failing.
• A safe repair/rebuild path is selected and validated on a copy or shadow collection before live mutation.
• The pre-repair backup/stop-daemon/operator sequence is documented.
• Normal KB/MC healthchecks remain bounded; any heavier exportability probe stays on-demand.

Out of Scope

• Repairing the FTS5 SQLite inverted index from #13467; that is a separate copy-validated path.
• Reopening #12450 based only on this evidence; current query canaries are healthy.
• Running live Chroma repair, REINDEX, VACUUM, collection delete/recreate, or MC rebuild without explicit operator authorization.
• Redesigning the REM pipeline or session-summary backlog; those remain under #12065 and related subs.

Avoided Traps

• Do not conflate queryability with exportability. Query can pass while stored-embedding materialization fails.
• Do not assume SQLite FTS5 rebuild fixes HNSW/stored-embedding retrieval. The copied FTS5 rebuild clears SQLite integrity, but it does not validate MC export.
• Do not trust a backup that silently skips corrupted vector IDs without surfacing counts and completeness.

Related

Related: #13467 Related: #12065 Related: #12450

Origin Session ID

Origin Session ID: c3a6e312-b858-4be4-ad97-9bc55cbad5ae

Handoff Retrieval Hints

• query_raw_memories: Memory Core Chroma stored embedding export Error finding id checkChromaIntegrity
• query_raw_memories: query canary healthy but collection.get include embeddings fails
• Exact commands: npm run ai:check-chroma-integrity -- --json --keep-snapshot; node ai/scripts/maintenance/probeCollectionQueryHealth.mjs