Make defrag repair resumable and lease-protected

Context

A live Memory Core repair-defrag run was started manually after Chroma corruption left thousands of metadata rows without stored vectors. The run re-embedded for more than five hours, reached roughly 8k regenerated embeddings, then hit an overcap embedding input and crashed. Restarting the script began from zero, which proves the current mutating repair path keeps regenerated vectors in process memory until the final promotion step instead of making successful work durable as it progresses.

A second defect surfaced in the same operator flow: ai/scripts/runners/runSandman.mjs and ai/scripts/maintenance/backup.mjs already acquire the shared heavy-maintenance lease, but standalone defragChromaDB.mjs does not. When defrag is started manually and the orchestrator starts afterward, the orchestrator cannot observe the already-active defrag as a heavy maintenance task and may schedule other heavy work on top.

Live latest-open sweep: checked latest 20 open issues on 2026-06-25. #13999 covers the earlier backup/exportability incident framing and #14017 covers timestamped progress logs, but no open ticket covers the combined defect: resumable repair-defrag plus standalone defrag heavy-maintenance lease publication. A2A in-flight sweep: checked latest 30 messages on 2026-06-25; only the prior @neo-gpt #13999 claim overlaps, with no competing peer claim for this combined defrag scope.

The Problem

The current repair-defrag behavior is unsafe for a 13k-row recovery because it is effectively one-shot. A provider-side overcap row, process crash, terminal interruption, or host restart can discard hours of already-successful re-embedding work.

The scheduling bug compounds that risk. The heavy-maintenance lease is the cross-process coordination primitive the orchestrator already respects for graph-heavy/manual maintenance. Defrag is a heavy maintenance operation by definition: it snapshots the Chroma persist dir, enumerates collections, writes shadow collections, promotes collection names, and can run for hours. If a standalone defrag does not hold the lease, the orchestrator has no durable signal to defer Sandman/dream/backup/defrag-class work while the manual defrag is active.

The Architectural Reality

• ai/scripts/maintenance/defragChromaDB.mjs owns the standalone defrag CLI and the Memory Core repair-defrag path.
• ai/scripts/maintenance/repairMemoryCoreStoredEmbeddings.mjs owns extraction/re-embedding of missing stored vectors.
• ai/daemons/orchestrator/services/HeavyMaintenanceLeaseService.mjs owns the cross-process heavy-maintenance lease primitive.
• ai/scripts/runners/runSandman.mjs is the canonical manual REM runner and already wraps its operation in withHeavyMaintenanceLease(...).
• ai/scripts/maintenance/backup.mjs already wraps manual backup execution in withHeavyMaintenanceLease(...).
• Knowledge Base evidence for defragChromaDB points to earlier hardening around unified-store-safe shadow promotion and durable phase markers in #12140/#12142/#12153; this ticket extends that hardening to the current repair-defrag incident.

The Fix

1. Make Memory Core repair-defrag durable by streaming recovered batches into a resumable shadow collection as they are extracted/re-embedded.
2. Record enough state for reruns to resume from the existing shadow collection instead of restarting provider work from zero: target, collection name, source count, shadow name, loaded count, phase, and unrecoverable preview.
3. Treat provider-overcap rows as per-row unrecoverable failures after single-document isolation, not as a process-killing exception that discards previous work.
4. Wrap standalone defragChromaDB.mjs CLI execution in withHeavyMaintenanceLease(...) with owner/reason metadata equivalent to backup and Sandman, so a manually started defrag is visible to the orchestrator before any other heavy task starts.
5. Preserve module-level/test callers' control: exported/internal defrag functions remain lease-free; only the auto-run CLI path acquires the shared lease.

Contract Ledger Matrix

Decision Record Impact

Aligned with the existing Agent OS heavy-maintenance lease design and ADR-0017 unified Chroma store hardening. No new ADR is required: this is an implementation correction to make defrag obey the existing cross-process maintenance mutex and durable-state model.

Acceptance Criteria

• Memory Core repair-defrag writes each recovered batch to a shadow collection before the next provider embedding call.
• Restarting after a crash/abort resumes from the shadow collection's existing IDs instead of re-embedding already-loaded rows.
• A single overcap embedding input is isolated and reported as unrecoverable without discarding successful prior rows.
• Standalone defragChromaDB.mjs auto-run acquires the shared heavy-maintenance lease with clear owner/reason metadata.
• If another heavy maintenance task already holds the lease, standalone defrag defers without touching Chroma.
• Unsafe promotion phases remain fail-closed and require manual recovery rather than blind resume.
• Focused unit coverage proves resumable repair, malformed resume rejection, batch streaming, overcap isolation, and standalone lease acquisition/deferral.

Out of Scope

• Do not redesign orchestrator scheduling or the lease service itself.
• Do not reopen or repurpose #13999 as the implementation authority for this combined scope.
• Do not run live defrag from CI or unit tests.
• Do not add progress timestamping; that remains #14017.

Avoided Traps

• A dry-run-only diagnostic is insufficient; the mutating repair path must make progress durable.
• A new defrag-specific lock would fork the scheduling contract; use the existing heavy-maintenance lease the orchestrator already understands.
• Swallowing overcap rows as success would corrupt repair semantics; report them as unrecoverable and abort promotion while preserving resumable shadow progress.

Related

• #13999 — earlier Memory Core backup/exportability incident context; not the implementation authority for this combined ticket.
• #14017 — timestamped progress logging follow-up, separate from durability/scheduling.
• #12140, #12142, #12153 — prior unified Chroma/defrag hardening context.

Retrieval Hint: defragChromaDB Memory Core repair resumable shadow heavy-maintenance lease standalone defrag orchestrator scheduling