PREMISE CORRECTED (v2, 2026-07-31): the version-incompatibility diagnosis below is falsified — host CLI 1.4.4 and Docker server 1.5.9 are the SAME Chroma release commit (CLI/server use separate version namespaces). The real root cause: compose mounts the named volume at /chroma/unified (and sets PERSIST_DIRECTORY accordingly) while the image's chroma run /config.yaml persists to /data — the volume is empty, the phantom collection IDs came from inspecting a different directory than the API served, and the completed rebuild currently lives in the container's UNMOUNTED writable layer. Full falsification: the root-cause correction comment below (2026-07-31T19:4xZ). Standing orders until the mount contract is fixed: no container remove/recreate; a warm /data snapshot is preserved at a host path (receipt on #16167); the authoritative preservation is a QUIESCED copy taken during the fix window.STANDING ORDER LIFTED 2026-08-05 — struck here 2026-08-09. It was scoped to a condition and the condition is met: the mount contract is fixed (chroma-data → /data, PERSIST_DIRECTORY=/data, /chroma/unified absent), verified on the running plane by me and independently by @neo-opus-grace and @neo-opus-ada. Recreate-survival is proven twice under today's compose (ACs below). No restriction on container remove/recreate remains.
Why this sat stale for four days: the lifting comment recorded that it could not strike the order in place — "no issue-body edit path available to me this turn" — and deferred to the comment as authoritative. That made the body's most operationally-loaded sentence contradict the ticket's real state, on a MUST row, where a reader following the body would refuse a recreate nobody was asking them to refuse. The constraint was a per-turn tooling gap, not a permanent one, and it was never re-checked once it lifted. A correction parked in a comment is not a correction; the body is the state. This ticket's remaining scope re-derives from the corrected premise: mount/command contract fix + recreate-survival proof + source-operation derivation for any backfill. — @neo-opus-vega (author), correction by @neo-gpt-emmy
Context
Found on 2026-07-31 completing the #16167 cutover of this machine to the container topology. The graph ported cleanly by bind mount — 183,373 nodes / 109,541 edges, byte-exact. The vector store did not.
ai/deploy/docker-compose.yml:25 pins chromadb/chroma:1.5.9. The 4.1 GB store at .neo-ai-data/chroma/unified was written by the host-side binary shipped with the npm chromadb package (node_modules/.bin/chroma, reporting 1.4.4). 1.5.9 cannot read that store's on-disk layout.
⚠️ That sentence is FALSIFIED and is retained only because it is this ticket's origin claim. It is now disproven three independent ways: release-commit namespaces (@neo-gpt-emmy, premise correction above), byte-identical schema ledgers on both stores (@neo-opus-grace, sysdb: 10 · metadb: 6 · embeddings_queue: 2), and — 2026-08-09 — a live 1.5.9 server serving records out of the legacy store: 21 databases resolved, 329be86b listed, count = 31,202, documents actually returned, zero errors and zero migration lines. The store was never exposed (source mounted :ro, copy in a throwaway volume; original verified unchanged at 2160865280 bytes, mtime Jul 31 06:41). Receipt: comment IC_kwDODSospM8AAAABN9Q6nQ.
The legacy store is therefore RECOVERABLE, not stranded — remaining migration work is a copy problem, not a rescue. And most of the 4.1 GB is test debris: 454 of 461 collections live in neo-unit-test, 20 of 21 databases are graph-service-test-*, leaving 7 real collections in default_database.
Live latest-open sweep at 2026-07-31T01:30Z: no open issue covers the Chroma server version or a vector-store migration. #16206 covers compose layering and is adjacent, not duplicative.
The Problem
It fails silently, in the worst possible direction. 1.5.9 starts, reports healthy, resolves the collection names, and serves fresh empty collections under them. The proof is the identity mismatch:
Source
neo-agent-memory id
count
Chroma 1.5.9 API
b419ba4f-3802-4c09-93cf-b8c63b99d1dd
0
The persisted store (chroma.sqlite3)
329be86b-242e-4047-88d1-4c672e94dd37
31,187
Chroma 1.4.4 API (the writer)
329be86b-242e-4047-88d1-4c672e94dd37
31,187
b419ba4f exists nowhere in the store — verified by direct SQLite read with no WAL present, so it is not a stale-snapshot artifact. 1.4.4 returns the store's own id. That is a version incompatibility, not a configuration error.Inverted by the premise correction: it is a configuration error, not a version incompatibility. The server was serving a different directory than the one being inspected — see the mount root cause above. The observed symptom was exactly right; the cause attributed to it was exactly backwards.
Under 1.5.9, query_raw_memories returned count: 0, results: []with no error, against 31,187 memory and 61,206 knowledge-base embeddings on disk. Memory Core reported status: healthy and "All features are operational".
That combination is the whole severity: a peer running a prior-art sweep gets an empty result and correctly concludes nothing exists. There is no failure to notice. And because MC reports healthy, it will happily write new embeddings into the shadow collections while the historical vectors sit orphaned in the same file — a split-brain that widens with every memory saved and is invisible until someone compares ids.
A related trap this ticket should record: the count: 0 initially looked like a symptom of a separate defect (the embedding provider was unreachable from the container — NEO_OPENAI_COMPATIBLE_HOST interpolated the host's 127.0.0.1:1234 loopback, tracked in #16206). Fixing that turned status from degraded to healthy but left count at 0. Two independent faults presented as one number. Anyone diagnosing this must not stop at the first cause that moves the status.
The Architectural Reality
chromadb/chroma:1.4.4 is not published. Verified against Docker Hub: 1.4.4 → HTTP 404, 1.4.9 → 404, while 1.4.0 → 200 and 1.5.0 → 200. So "just pin the writer's version" is not available as a container image.
1.4.0 would be a downgrade from the writer. Patch-level regressions are exactly where a schema migration can be destructive, so pinning below 1.4.4 trades a read failure for a write risk.
Client and server versions are separately numbered and easy to conflate.package.json:208 pins chromadb: ^3.5.0 (the JS client); that package's bundled CLI reports 1.4.4; the images use yet another line (1.5.x, currently publishing 1.5.10.dev*). Nothing in the repo states which server version owns the store format.
The store is the classic single-node layout — segment types urn:chroma:segment/vector/hnsw-local-persisted and urn:chroma:segment/metadata/sqlite, sysdb migrations at 10, metadb at 6, embeddings_queue at 2. 461 collections, 119,512 embeddings, 21 databases (mostly graph-service-test-* residue).
Current stopgap — deliberate, and the reason this ticket exists
Chroma runs host-side at 1.4.4, the exact binary that wrote the store, bound 0.0.0.0:8000 so containers can reach it. Containerized MC/KB point at it via NEO_CHROMA_HOST=host.docker.internal, with depends_on on the chroma container removed.
Verified working: query_raw_memories returns real records, and healthcheck reports memories=31187 summaries=1568.
The stopgap's cost is exactly what makes it temporary:
The cutover is partial. MC and KB are containerized; the vector store is not. The plane is not actually in Docker.
It pins an unpublished, superseded server version as load-bearing infrastructure, with no upgrade path recorded.
It lives in an untracked local override, so it is invisible to anyone else provisioning this topology.
A full clone of the store was taken before any version was pointed at it: /Users/Shared/neo-chroma-preflip-20260731-031207/unified (4.1 GB, chroma.sqlite3 byte-identical at 2,160,865,280 bytes).
The Fix (v4.1 — concept-passed; authoritative sequence in comments 5150971936 → 5150984731 → 5151051356)
The premise correction (same release commit; the image persists to /data while compose mounted /chroma/unified) retired the migration/re-embed framing this section originally carried. The mount contract is #16254, delivered by PR #16252 (related, non-closing). This ticket owns the execution:
Quiesce MC + KB + orchestrator; prove Chroma request-idle. Window trigger (operator-delegated to the team): 2b wake receipt + PR #16252 merged.
Physically seed the named volume from that copy before first boot (mount PR merged first).
Chroma-service-only recreate; prove M0 exact — persistence established without changing the dataset.
Docker-target-bound ai:restore --mode merge --only-substrate=kb,mc from the 07-30 bundle (KB 61,206; MC 31,173 + 1,554); per-substrate inserted/skipped receipt. Never the bare host command against the retired host plane.
Legacy-tail proof L ⊆ M0 ∪ B; one legacy-sourced logical delta merge if uncovered (+29 memories / +14 sessions known). 6b. native-graph receipt: re-projection from the SQLite SSOT via the existing graphProjection/heal machinery (fallback: legacy logical merge); reconciled count + one known-hit GP query.
M1 capture; second chroma-only recreate proving M1 exact; services up; known-hit MC/KB/GP queries; fresh backup with a clean integrity verdict.
Delivered by the v4.1 execution window 2026-08-01 11:48Z → 12:10Z and its closeout; the numbered receipts live in those two comments.
Window receipt: opened on the two-gate trigger; MC/KB/orchestrator stopped; Chroma proven request-idle. — receipt 1: stopped 11:51:08Z, zero Chroma log lines through a 25s settle.
Legacy-tail proof: L ⊆ M0 ∪ B holds (or the delta was merged and the proof re-run); legacy deletion gated on it. — receipt 6: legacyOnly = 0 on memory/sessions/KB/temporal. The sole tail (native-graph 1,144) was merged in the closeout and the proof re-run to legacyOnly = 0. Deletion is data-unblocked; the decision stays the operator's.
Recreate-survival #2: M1 exact; services healthy; known-hit MC and KB queries return; fresh backup integrity verdict clean. — receipt 7. Both sides of this recreate ran under today's compose — recreate #1 had already produced the chroma-data → /data container at the merged mount fix 247dbbc844 — so this is the steady-state chroma-only proof, not a migration-crossing one. Backup backup-2026-08-01T12-13-23.398Z, inline row-count parity clean; its overall status reads failed on the required-off-host-sync gate, disabled in this container profile (#16199 fail-loud behaving correctly, not a bundle defect).
native-graph: reconciled count + one known-hit GP query. — both halves delivered. — reconciled count delivered (385 → 1,529, legacyOnly = 0, count == legacy exactly; graph node search serves live). Open only for the full GP-synthesis known-hit, which rides a golden-path lane run on the #16256-rebuilt image.
WAL continuity: add_memory writes accepted before the window drain cleanly after it — no losses across the interval. — post-window WAL depth 0, and 5 organic live writes landed on top of M1. Neither is the stated property: no individual pre-window-accepted write has been traced across the interval. That trace is the remaining probe.
Whether the current volume was ever written by Chroma 1.4.4 — answered: NO, by @neo-opus-grace, reproduced independently by @neo-opus-vega. The 1.4.4 writer's collection 329be86b is absent from the current volume (6 collections, neo-agent-memory carries a different id) and present in the legacy host store (461 collections). Two separate stores. Stated bound: this proves no 1.4.4-written collection survives in the current volume; it cannot exclude a write-then-delete history, which no on-disk artifact would show.
Out of Scope
The 461 collections / 21 databases of graph-service-test-* residue, unless step 3's re-embed sheds it as a side effect. Vacuuming it is its own leaf.
NEO_OPENAI_COMPATIBLE_HOST container-loopback interpolation — separate fault, tracked in #16206.
Compose layering and fragment decomposition — #16206.
The graph SQLite, which ported correctly and needs nothing.
Avoided Traps
Stopping at the first cause that moves the status. The unreachable embedding provider and the unreadable store both presented as count: 0. Fixing the first flipped degraded → healthy and changed nothing about recall.
Trusting a count without checking identity.count: 0 is what a working server returns for a genuinely empty collection. Only the served-vs-persisted id comparison distinguishes "empty" from "wrong collection".
Pinning 1.4.0 because 1.4.4 is unavailable. Below the writer's version is a downgrade, and downgrades are where migrations destroy data.
Treating status: healthy as evidence. It reported healthy while serving an empty shadow of a 4.1 GB store.
Migrating the live store first. A clone exists specifically so the irreversible step is rehearsable.
Assuming the bind proved the migration. Four verification axes (bytes, nodes, edges, directory size) all passed while semantic recall was dead, because every one of them measured SQLite or file size and none exercised a query.
Decision Record impact
ADR 0017 §2.2 is corrected in place by #16254 / PR #16252: the persist path is image-owned (/data), parity is logical/topological (unified as the store's logical identity), and the falsified literal-leaf mechanism is retained only as a dated falsification record. ADR 0003's one-daemon / one-flat-store decision is unaffected.
Related
#16254 / PR #16252 — the compose mount contract (delivered separately; related, non-closing here)
#16167 — the cutover that surfaced it; the window trigger rides its 2b wake receipt
#16206 — ai/deploy compose layering; owns the NEO_OPENAI_COMPATIBLE_HOST interpolation defect found alongside this
#16205 — provisioning templates still emitting the retired stdio shape
ADR 0003 / ADR 0017, Epic #12153 — the unified Chroma store
Why this sat stale for four days: the lifting comment recorded that it could not strike the order in place — "no issue-body edit path available to me this turn" — and deferred to the comment as authoritative. That made the body's most operationally-loaded sentence contradict the ticket's real state, on a MUST row, where a reader following the body would refuse a recreate nobody was asking them to refuse. The constraint was a per-turn tooling gap, not a permanent one, and it was never re-checked once it lifted. A correction parked in a comment is not a correction; the body is the state. This ticket's remaining scope re-derives from the corrected premise: mount/command contract fix + recreate-survival proof + source-operation derivation for any backfill. — @neo-opus-vega (author), correction by @neo-gpt-emmy
Context
Found on 2026-07-31 completing the #16167 cutover of this machine to the container topology. The graph ported cleanly by bind mount — 183,373 nodes / 109,541 edges, byte-exact. The vector store did not.
ai/deploy/docker-compose.yml:25pinschromadb/chroma:1.5.9. The 4.1 GB store at.neo-ai-data/chroma/unifiedwas written by the host-side binary shipped with the npmchromadbpackage (node_modules/.bin/chroma, reporting 1.4.4).1.5.9 cannot read that store's on-disk layout.⚠️ That sentence is FALSIFIED and is retained only because it is this ticket's origin claim. It is now disproven three independent ways: release-commit namespaces (@neo-gpt-emmy, premise correction above), byte-identical schema ledgers on both stores (@neo-opus-grace,
sysdb: 10 · metadb: 6 · embeddings_queue: 2), and — 2026-08-09 — a live 1.5.9 server serving records out of the legacy store: 21 databases resolved,329be86blisted,count= 31,202, documents actually returned, zero errors and zero migration lines. The store was never exposed (source mounted:ro, copy in a throwaway volume; original verified unchanged at2160865280bytes, mtimeJul 31 06:41). Receipt: commentIC_kwDODSospM8AAAABN9Q6nQ.The legacy store is therefore RECOVERABLE, not stranded — remaining migration work is a copy problem, not a rescue. And most of the 4.1 GB is test debris: 454 of 461 collections live in
neo-unit-test, 20 of 21 databases aregraph-service-test-*, leaving 7 real collections indefault_database.Live latest-open sweep at 2026-07-31T01:30Z: no open issue covers the Chroma server version or a vector-store migration. #16206 covers compose layering and is adjacent, not duplicative.
The Problem
It fails silently, in the worst possible direction. 1.5.9 starts, reports healthy, resolves the collection names, and serves fresh empty collections under them. The proof is the identity mismatch:
neo-agent-memoryidb419ba4f-3802-4c09-93cf-b8c63b99d1ddchroma.sqlite3)329be86b-242e-4047-88d1-4c672e94dd37329be86b-242e-4047-88d1-4c672e94dd37b419ba4fexists nowhere in the store — verified by direct SQLite read with no WAL present, so it is not a stale-snapshot artifact. 1.4.4 returns the store's own id.That is a version incompatibility, not a configuration error.Inverted by the premise correction: it is a configuration error, not a version incompatibility. The server was serving a different directory than the one being inspected — see the mount root cause above. The observed symptom was exactly right; the cause attributed to it was exactly backwards.Under 1.5.9,
query_raw_memoriesreturnedcount: 0, results: []with no error, against 31,187 memory and 61,206 knowledge-base embeddings on disk. Memory Core reportedstatus: healthyand"All features are operational".That combination is the whole severity: a peer running a prior-art sweep gets an empty result and correctly concludes nothing exists. There is no failure to notice. And because MC reports healthy, it will happily write new embeddings into the shadow collections while the historical vectors sit orphaned in the same file — a split-brain that widens with every memory saved and is invisible until someone compares ids.
A related trap this ticket should record: the
count: 0initially looked like a symptom of a separate defect (the embedding provider was unreachable from the container —NEO_OPENAI_COMPATIBLE_HOSTinterpolated the host's127.0.0.1:1234loopback, tracked in #16206). Fixing that turnedstatusfromdegradedtohealthybut leftcountat 0. Two independent faults presented as one number. Anyone diagnosing this must not stop at the first cause that moves the status.The Architectural Reality
chromadb/chroma:1.4.4is not published. Verified against Docker Hub:1.4.4→ HTTP 404,1.4.9→ 404, while1.4.0→ 200 and1.5.0→ 200. So "just pin the writer's version" is not available as a container image.package.json:208pinschromadb: ^3.5.0(the JS client); that package's bundled CLI reports 1.4.4; the images use yet another line (1.5.x, currently publishing1.5.10.dev*). Nothing in the repo states which server version owns the store format.urn:chroma:segment/vector/hnsw-local-persistedandurn:chroma:segment/metadata/sqlite,sysdbmigrations at 10,metadbat 6,embeddings_queueat 2. 461 collections, 119,512 embeddings, 21 databases (mostlygraph-service-test-*residue).Current stopgap — deliberate, and the reason this ticket exists
Chroma runs host-side at 1.4.4, the exact binary that wrote the store, bound
0.0.0.0:8000so containers can reach it. Containerized MC/KB point at it viaNEO_CHROMA_HOST=host.docker.internal, withdepends_onon the chroma container removed.Verified working:
query_raw_memoriesreturns real records, andhealthcheckreportsmemories=31187 summaries=1568.The stopgap's cost is exactly what makes it temporary:
A full clone of the store was taken before any version was pointed at it:
/Users/Shared/neo-chroma-preflip-20260731-031207/unified(4.1 GB,chroma.sqlite3byte-identical at 2,160,865,280 bytes).The Fix (v4.1 — concept-passed; authoritative sequence in comments 5150971936 → 5150984731 → 5151051356)
The premise correction (same release commit; the image persists to
/datawhile compose mounted/chroma/unified) retired the migration/re-embed framing this section originally carried. The mount contract is #16254, delivered by PR #16252 (related, non-closing). This ticket owns the execution:/datacopy + file-tree checksums.ai:restore --mode merge --only-substrate=kb,mcfrom the 07-30 bundle (KB 61,206; MC 31,173 + 1,554); per-substrate inserted/skipped receipt. Never the bare host command against the retired host plane.graphProjection/heal machinery (fallback: legacy logical merge); reconciled count + one known-hit GP query.Contract Ledger
add_memorydown for its durationlearn/agentos/tooling/RestorationRunbook.mdai:restore --mode mergeneo-native-graphis the SQLite graph's projectionAcceptance Criteria
Delivered by the v4.1 execution window 2026-08-01 11:48Z → 12:10Z and its closeout; the numbered receipts live in those two comments.
/datacopy + checksums taken quiesced. — receipt 2:memory 23,862 · sessions 1,656 · KB 0 · native-graph 385 · temporal 69, per-collection sorted-ID sha256, 1.0G quiesced copy + manifest.legacyOnly = 0on memory/sessions/KB/temporal. The sole tail (native-graph 1,144) was merged in the closeout and the proof re-run tolegacyOnly = 0. Deletion is data-unblocked; the decision stays the operator's.chroma-data → /datacontainer at the merged mount fix247dbbc844— so this is the steady-state chroma-only proof, not a migration-crossing one. Backupbackup-2026-08-01T12-13-23.398Z, inline row-count parity clean; its overall status reads failed on the required-off-host-sync gate, disabled in this container profile (#16199 fail-loud behaving correctly, not a bundle defect).legacyOnly = 0, count == legacy exactly; graph node search serves live). Open only for the full GP-synthesis known-hit, which rides a golden-path lane run on the #16256-rebuilt image.add_memorywrites accepted before the window drain cleanly after it — no losses across the interval. — post-window WAL depth 0, and 5 organic live writes landed on top of M1. Neither is the stated property: no individual pre-window-accepted write has been traced across the interval. That trace is the remaining probe.329be86bis absent from the current volume (6 collections,neo-agent-memorycarries a different id) and present in the legacy host store (461 collections). Two separate stores. Stated bound: this proves no 1.4.4-written collection survives in the current volume; it cannot exclude a write-then-delete history, which no on-disk artifact would show.Out of Scope
graph-service-test-*residue, unless step 3's re-embed sheds it as a side effect. Vacuuming it is its own leaf.NEO_OPENAI_COMPATIBLE_HOSTcontainer-loopback interpolation — separate fault, tracked in #16206.Avoided Traps
count: 0. Fixing the first flippeddegraded→healthyand changed nothing about recall.count: 0is what a working server returns for a genuinely empty collection. Only the served-vs-persisted id comparison distinguishes "empty" from "wrong collection".status: healthyas evidence. It reported healthy while serving an empty shadow of a 4.1 GB store.Decision Record impact
ADR 0017 §2.2 is corrected in place by #16254 / PR #16252: the persist path is image-owned (
/data), parity is logical/topological (unifiedas the store's logical identity), and the falsified literal-leaf mechanism is retained only as a dated falsification record. ADR 0003's one-daemon / one-flat-store decision is unaffected.Related
ai/deploycompose layering; owns theNEO_OPENAI_COMPATIBLE_HOSTinterpolation defect found alongside thisOrigin Session ID: 0a7f5f1d-cf12-4698-984c-17b64eea5178
Retrieval Hint:
chroma 1.5.9 cannot read 1.4.4 store phantom collection id b419ba4f 329be86b count 0 silent empty semantic recall dead host-side stopgap host.docker.internal unpublished image tag forward migration re-embed from graph