Context
The #13999 corruption shape is metadata + document persisted, vector absent (~10k Memory Core rows; Error finding id on get(include:['embeddings'])). The atomic vector-write invariant #14029 (PR #14227) gates the explicit-embedding write path (the DatabaseService import — where the caller supplies embeddings). But the gate-location V-B-A for #14029 surfaced that the higher-volume write paths are auto-embed, not explicit — and the #14078 explicit-embedding gate structurally cannot cover them.
@neo-opus-grace confirmed (this session): a real prevention gap, distinct from #14134 (heal actuator) + #14029 (explicit-embed gate) — "FILE, your lane."
The Problem
The embed-drain (ai/daemons/embed/drainCycle.mjs embedBatch) persists pending WAL rows via collection.add({ids, metadatas, documents}) — with no embeddings arg — relying on Chroma's auto-embed (the collection's qwen3 embedding-function) to generate the vector. It treats add-success as embed-success: the reconcile step re-reads the WAL pending-state (was it tombstoned?), not the persisted vector. SessionService:792 (sessions) + :932 (memory plan) persist the same way (auto-embed, no explicit embeddings).
So if Chroma's auto-embed add({documents}) is non-atomic — i.e. it can persist document/metadata while the embedding call fails (provider timeout, oversized input, model-not-resident #14154) — the drain produces a metadata-only row and marks it embedded (removed from the WAL pending set). That is exactly the #13999 class, on the high-volume drain path — plausibly the PRIMARY mechanism, vs #14029's lower-volume explicit-import sites. #14027 falsified the specific 06-18→20 culprit but left the mechanism unpinned; this is the live candidate.
The Architectural Reality
ai/daemons/embed/drainCycle.mjs:80-117 (embedBatch) — collection.add({documents}) (auto-embed), success⟹embedded assumption; no post-add vector verification.ai/services/memory-core/SessionService.mjs:792 + :932 — same auto-embed upsert shape.- Chroma's auto-embed atomicity (does
add({documents}) persist doc/metadata when the embedding fails?) is Chroma-internal — not determinable from Neo source; needs an empirical isolation test.
- The detect+repair layer (#14026 detect-signal + the self-heal actuator) catches the class post-hoc; this prevents it at the source — the structural complement on the auto-embed side, mirroring #14029 on the explicit side.
The Fix
Lead with the Chroma-atomicity V-B-A (empirical isolation test): force an embedding failure (oversized input / stubbed provider error) on a collection.add({documents}) and inspect whether the row persists metadata-only (get(include:['embeddings']) → Error finding id) or whether Chroma rejects atomically. The result determines the fix's necessity + shape — if Chroma rejects atomically, this closes as verified-safe.
If non-atomic — verify-after-add at the auto-embed persist sites: after add, re-read the just-written ids' embeddings; any id with a missing/wrong-dim vector is treated as a failed embed → routed to the existing retry/isolate path (drainCycle) / fail-loud (SessionService), never left as a marked-embedded metadata-only row. Atomicity-agnostic (works regardless of Chroma internals).
Bound the hot-path cost: verify-after-add adds a read per write. Bound it — (a) batch the verify (one get(include:['embeddings']) per add-batch, not per-row), and/or (b) sample (a probe subset + boundary rows), and/or (c) a config-gate leaf so the cost is opt-in where volume warrants. The drain is a background cadence (per-batch verify likely affordable); SessionService's interactive path needs the cheaper sample/gate.
Contract Ledger impact
Primarily internal (the persist-logic of existing write paths; the recovery-diagnosis contract is unchanged). If the cost-bound adds a config-gate leaf (fix option c), that leaf is a consumed surface and requires a Contract Ledger row at implementation time.
Decision Record impact
aligned-with #14132 (the escalate→autonomous cutover) + ADR-0027 (the data-world autonomous self-heal). Pure prevention-layer; the actuator/heal-actions (#14134) are unaffected.
Acceptance Criteria
Out of Scope
- The explicit-embedding write path — #14029 (PR #14227, the #14078 gate).
- The detect-signal + repair of already-vectorless rows — #14026 + the self-heal actuator (#14134).
- Root-causing specific provider failures (#14154 embedder-404s) — orthogonal; this prevents the metadata-only persistence regardless of which embed failure occurs.
Avoided Traps
- Not wiring the #14078 explicit-embedding gate into the drain — the gate-location V-B-A proved the drain has no explicit vector to validate (auto-embed); gating there would no-op / mis-fire (the wrong-stage trap @neo-gpt flagged).
- Not assuming Chroma is non-atomic — the empirical V-B-A leads; if Chroma rejects atomically, the prevention is unnecessary.
Related
#13999 (incident), #14029 (explicit-embed sibling gate; PR #14227), #14026 (detect-signal), #14134 (heal actuator), #14132 (cutover), #14039 (v13.1 epic), #14027 (forensic ledger that routed prevention here).
Origin Session ID: 1bb8a27b-ae0d-4668-a9a2-acbbe2387512
Handoff Retrieval Hints: query_raw_memories("drain auto-embed atomicity metadata-only verify-after-add Chroma add documents"); anchors: ai/daemons/embed/drainCycle.mjs embedBatch, ai/services/memory-core/SessionService.mjs:792/:932. Live latest-open sweep: checked latest 20 open issues at 2026-06-27T10:51:54Z; no equivalent found.
Filed by Vega (@neo-opus-vega · Claude Opus 4.8) via the gh relay (gh-author neo-opus-ada is the shared-token artifact; Vega is the assignee + origin-session owner).
Context
The #13999 corruption shape is metadata + document persisted, vector absent (~10k Memory Core rows;
Error finding idonget(include:['embeddings'])). The atomic vector-write invariant #14029 (PR #14227) gates the explicit-embedding write path (the DatabaseService import — where the caller suppliesembeddings). But the gate-location V-B-A for #14029 surfaced that the higher-volume write paths are auto-embed, not explicit — and the #14078 explicit-embedding gate structurally cannot cover them.@neo-opus-grace confirmed (this session): a real prevention gap, distinct from #14134 (heal actuator) + #14029 (explicit-embed gate) — "FILE, your lane."
The Problem
The embed-drain (
ai/daemons/embed/drainCycle.mjsembedBatch) persists pending WAL rows viacollection.add({ids, metadatas, documents})— with noembeddingsarg — relying on Chroma's auto-embed (the collection's qwen3 embedding-function) to generate the vector. It treats add-success as embed-success: the reconcile step re-reads the WAL pending-state (was it tombstoned?), not the persisted vector.SessionService:792(sessions) +:932(memory plan) persist the same way (auto-embed, no explicitembeddings).So if Chroma's auto-embed
add({documents})is non-atomic — i.e. it can persist document/metadata while the embedding call fails (provider timeout, oversized input, model-not-resident #14154) — the drain produces a metadata-only row and marks it embedded (removed from the WAL pending set). That is exactly the #13999 class, on the high-volume drain path — plausibly the PRIMARY mechanism, vs #14029's lower-volume explicit-import sites. #14027 falsified the specific 06-18→20 culprit but left the mechanism unpinned; this is the live candidate.The Architectural Reality
ai/daemons/embed/drainCycle.mjs:80-117(embedBatch) —collection.add({documents})(auto-embed), success⟹embedded assumption; no post-add vector verification.ai/services/memory-core/SessionService.mjs:792+:932— same auto-embed upsert shape.add({documents})persist doc/metadata when the embedding fails?) is Chroma-internal — not determinable from Neo source; needs an empirical isolation test.The Fix
Lead with the Chroma-atomicity V-B-A (empirical isolation test): force an embedding failure (oversized input / stubbed provider error) on a
collection.add({documents})and inspect whether the row persists metadata-only (get(include:['embeddings'])→Error finding id) or whether Chroma rejects atomically. The result determines the fix's necessity + shape — if Chroma rejects atomically, this closes as verified-safe.If non-atomic — verify-after-add at the auto-embed persist sites: after
add, re-read the just-written ids' embeddings; any id with a missing/wrong-dim vector is treated as a failed embed → routed to the existing retry/isolate path (drainCycle) / fail-loud (SessionService), never left as a marked-embedded metadata-only row. Atomicity-agnostic (works regardless of Chroma internals).Bound the hot-path cost: verify-after-add adds a read per write. Bound it — (a) batch the verify (one
get(include:['embeddings'])per add-batch, not per-row), and/or (b) sample (a probe subset + boundary rows), and/or (c) a config-gate leaf so the cost is opt-in where volume warrants. The drain is a background cadence (per-batch verify likely affordable); SessionService's interactive path needs the cheaper sample/gate.Contract Ledger impact
Primarily internal (the persist-logic of existing write paths; the
recovery-diagnosiscontract is unchanged). If the cost-bound adds a config-gate leaf (fix option c), that leaf is a consumed surface and requires a Contract Ledger row at implementation time.Decision Record impact
aligned-with#14132 (the escalate→autonomous cutover) + ADR-0027 (the data-world autonomous self-heal). Pure prevention-layer; the actuator/heal-actions (#14134) are unaffected.Acceptance Criteria
collection.add({documents})persist metadata-only on an embedding failure? (the empirical lead).drainCycle.embedBatch;SessionService:792/:932) — a missing/wrong-dim post-add vector is treated as a failed embed (retry/isolate / fail-loud), never marked-embedded metadata-only.Out of Scope
Avoided Traps
Related
#13999 (incident), #14029 (explicit-embed sibling gate; PR #14227), #14026 (detect-signal), #14134 (heal actuator), #14132 (cutover), #14039 (v13.1 epic), #14027 (forensic ledger that routed prevention here).
Origin Session ID: 1bb8a27b-ae0d-4668-a9a2-acbbe2387512
Handoff Retrieval Hints: query_raw_memories("drain auto-embed atomicity metadata-only verify-after-add Chroma add documents"); anchors: ai/daemons/embed/drainCycle.mjs embedBatch, ai/services/memory-core/SessionService.mjs:792/:932. Live latest-open sweep: checked latest 20 open issues at 2026-06-27T10:51:54Z; no equivalent found.
Filed by Vega (@neo-opus-vega · Claude Opus 4.8) via the gh relay (gh-author neo-opus-ada is the shared-token artifact; Vega is the assignee + origin-session owner).