Operator-reported P0 (2026-06-26). A KB sync failed at embedding batch 214 of 328 (Failed to process batch 214 after 5 retries. Aborting.) and lost all 213 completed batches — the re-run restarts from batch 1, re-embedding all 16382 chunks. A ~30-second provider blip cost the entire corpus re-embed.
Root cause of the data-loss (Neo-side, independent of what tripped batch 214)
VectorService.embedViaShadowSwap is all-or-nothing. It builds the entire shadow collection across every batch and only promotes on full success. A single failed batch — embedChunks throw at VectorService.mjs:716 — is caught at line 744, the incomplete shadow is parked/discarded (parkFailedShadowCollection), and the next run rebuilds from scratch. So any single transient embedding-batch failure discards all completed embedding progress, plus leaves a window with no refreshed KB.
This is the amplifier that turns a transient provider hiccup into a P0: the trigger (one batch's The requested resource could not be found) is recoverable-in-principle, but the swap design converts it into total-progress-loss.
Fix direction (one or more)
- Checkpoint / resume — persist embedded-batch progress so a re-run resumes from the last completed batch. The shadow collection already holds the completed batches at failure time; resume into it instead of discarding + rebuilding from batch 1.
- Resilient batch handling — a bounded transient batch failure should retry/skip-and-record without aborting the whole shadow-swap; abort only on a systemic/persistent failure (mirrors the v13.1 systemic-fault-bound discipline).
- Recognize provider 404 in the retry guard —
TextEmbeddingService.#postOpenAiCompatible's isModelLoadError only matches HTTP-400 Model was unloaded / Failed to load model shapes; an HTTP-404 gets 5 blind re-POSTs then aborts. Sub-fix (the trigger's true cause is provider-side per the incident).
Acceptance
Related
- Trigger was a provider 404. Config-hygiene cleanup (separate):
openAiCompatible.host defaults to http://127.0.0.1:11434 (Ollama's port) while LM Studio runs on :1234 (= orchestrator.lms.port) — the tracked default points at the wrong port for an LMS deployment.
- Part of #14039 (v13.1 Agent OS Stability / data-integrity immune system) — this is the KB-sync analogue of the Memory Core data-recovery work.
Authored by Grace (Claude Opus 4.8, Claude Code).
Operator-reported P0 (2026-06-26). A KB sync failed at embedding batch 214 of 328 (
Failed to process batch 214 after 5 retries. Aborting.) and lost all 213 completed batches — the re-run restarts from batch 1, re-embedding all 16382 chunks. A ~30-second provider blip cost the entire corpus re-embed.Root cause of the data-loss (Neo-side, independent of what tripped batch 214)
VectorService.embedViaShadowSwapis all-or-nothing. It builds the entire shadow collection across every batch and only promotes on full success. A single failed batch —embedChunksthrow atVectorService.mjs:716— is caught at line 744, the incomplete shadow is parked/discarded (parkFailedShadowCollection), and the next run rebuilds from scratch. So any single transient embedding-batch failure discards all completed embedding progress, plus leaves a window with no refreshed KB.This is the amplifier that turns a transient provider hiccup into a P0: the trigger (one batch's
The requested resource could not be found) is recoverable-in-principle, but the swap design converts it into total-progress-loss.Fix direction (one or more)
TextEmbeddingService.#postOpenAiCompatible'sisModelLoadErroronly matches HTTP-400Model was unloaded/Failed to load modelshapes; an HTTP-404 gets 5 blind re-POSTs then aborts. Sub-fix (the trigger's true cause is provider-side per the incident).Acceptance
Related
openAiCompatible.hostdefaults tohttp://127.0.0.1:11434(Ollama's port) while LM Studio runs on:1234(=orchestrator.lms.port) — the tracked default points at the wrong port for an LMS deployment.Authored by Grace (Claude Opus 4.8, Claude Code).