Promote recoverable Memory Core repair rows

Context

The live Memory Core repair/defrag work recovered a large shadow corpus but still aborts collection promotion when a single row remains unrecoverable. Vega flagged this as finding-b from the #14027 forensic lane: a multi-hour repair can produce tens of thousands of recovered rows and still leave canonical Chroma unrepaired because one bad row blocks promotion.

Live latest-open sweep: checked latest 30 open issues at 2026-06-26T05:11:56Z. Closest tickets were #14023 (unrecoverable reason metadata while explicitly preserving any-unrecoverable-blocks-promotion), #14027 (forensic vector-loss audit), #14029 (write-path atomic vector invariant), and #14030 (backup reliability). No equivalent ticket for partial promotion / unrecoverable quarantine existed.

Targeted GitHub search defrag unrecoverable promote recoverable shadow rows repair all-or-nothing returned no open issues. A2A in-flight sweep over the latest 30 messages found Vega's MESSAGE:5e8ef670-d6f1-446d-8e1b-14bc96306c24, which explicitly delegated finding-b disposition to @neo-gpt, and no competing lane claim. KB/local sweep found the current all-or-nothing contract in #14023, #14027, and defragChromaDB.mjs, but no sibling implementation ticket for changing promotion semantics.

The Problem

Current repair behavior is fail-loud, but too coarse for the incident class we are recovering from. One unrecoverable row causes the mutating repair to abort before shadow promotion, leaving all recovered rows unavailable in canonical Memory Core. In the live incident, that means a 5h repair can re-embed most of the corpus, then canonical remains corrupted and a restart can force expensive work to restart from too little durable progress.

We still need fail-loud semantics. The wrong fix would be to silently drop unrecoverable rows and report success. The right shape is a fail-loud partial-repair contract: promote or finalize the recoverable majority only when the unrecoverable minority is explicitly quarantined/manifests as data loss, keep the pre-nuke/parked source available, and surface a non-green status that operators and CI cannot mistake for a clean repair.

The Architectural Reality

• ai/scripts/maintenance/repairMemoryCoreStoredEmbeddings.mjs:113-177 classifies missing-vector rows as recoverable when documents can be re-embedded, and appends ids to unrecoverable when metadata/documents are absent or per-document embedding isolation fails.
• ai/scripts/maintenance/defragChromaDB.mjs:939-990 streams recoverable rows into a resumable shadow collection, but if unrecoverable.length > 0 it writes phase: 'memory-core-repair-aborted' and returns before promotion.
• ai/scripts/maintenance/defragChromaDB.mjs:1175-1178 stops the per-collection repair loop on the first aborted collection.
• ai/scripts/maintenance/defragChromaDB.mjs:1238-1248 defines anyRepairAborted() as the operator-facing fail-loud predicate, so an aborted collection is never reported as successful repair.
• #14023 deliberately preserves the current safety contract: any unrecoverable entry still blocks promotion. This ticket is the successor that changes that behavior only after quarantine/manifest semantics exist.
• #14027 current evidence shows an active repair shadow successfully re-embedding sampled old canonical vector failures while the canonical collection remains corrupted until promotion.

The Fix

Introduce a bounded partial-repair finalization path for Memory Core repair-defrag:

1. Define an explicit partial-promotion mode/result such as partial-promoted or promoted-with-unrecoverables, distinct from clean success and distinct from abort.
2. When recoverable rows are loaded into the repair shadow and only bounded unrecoverables remain, promote/finalize the recovered shadow through the existing shadow/parking safety envelope while preserving the original source via pre-nuke snapshot/parked collection.
3. Write a durable unrecoverable manifest with ids/reasons/counts that downstream diagnostics can inspect. If #14023 has landed, consume its structured unrecoverable entries; otherwise keep the implementation compatible with ids-only unrecoverables while leaving the manifest schema ready for reasons.
4. Keep fail-loud semantics at the CLI boundary: the command must not report a clean repair when unrecoverables remain. The outcome should be distinguishable from both success and aborted, and should exit/report in a way operators cannot ignore.
5. Add focused unit coverage for clean promotion, aborted repair, and partial promotion/quarantine behavior, including state-marker lifecycle and operator-visible output.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR-0025 / ADR-0026 self-healing safety: this is a data-recovery actuator slice that preserves explicit diagnosis, bounded action, and fail-loud operator visibility. No accepted ADR is amended. It does challenge the local #14023 assumption that any unrecoverable row must always block promotion; that challenge is scoped to a successor ticket with a manifest/quarantine safety envelope.

Acceptance Criteria

• Memory Core repair can produce a distinct partial-promotion outcome when recoverable rows are promotion-ready and unrecoverable rows remain bounded/manifested.
• Partial promotion never reports clean success; operator output and returned results distinguish clean, partial, and aborted.
• A durable unrecoverable manifest is written with ids and, when available, reason metadata.
• Existing abort behavior remains available when safety preconditions fail or partial mode is disabled.
• State-marker lifecycle prevents the next run from restarting from zero when a resumable/partial repair state already exists.
• Focused unit coverage exercises clean promotion, fail-loud abort, partial promotion, manifest writing, and CLI summary/exit semantics.

Out of Scope

• Reopening or reverting #14021.
• Replacing the whole repair-defrag architecture.
• Running live mutating defrag in CI.
• Solving the original 06-18 to 06-20 vector-loss root cause; that remains #14027 / #14029 dependent.
• Weakening fail-loud behavior into silent row drops.

Avoided Traps

• Do not call partial promotion a clean success. The operator must see that unrecoverables remain.
• Do not delete or hide the original source before the manifest and parked/pre-nuke recovery path are durable.
• Do not block all recovered rows forever just to preserve a mathematically pure all-or-nothing contract. In incident recovery, that turns one unrecoverable row into full-corpus operational failure.
• Do not duplicate #14023 reason work; consume it if available, or keep this manifest compatible with ids-only entries.

Related

Refs #14039

Related: #14027, #14023, #14021, #13999, #14029

Origin Session ID: 35f83031-f1a6-41a7-9c3b-089b87307db9

Handoff Retrieval Hints: query_raw_memories("#14027 finding-b repair all-or-nothing no promotion unrecoverable shadow"); query_raw_memories("Memory Core repair partial promotion unrecoverable manifest defrag"); source anchors repairMemoryCoreCollectionViaResumableShadow, repairMemoryCoreCollectionsViaFullEnumeration, anyRepairAborted, extractMemoryCoreCollectionData.