LearnNewsExamplesServices
Frontmatter
id14146
titleP0: KB-sync shadow-swap discards ALL embedding progress on one failed batch — no checkpoint/resume
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtJun 26, 2026, 11:26 PM
updatedAtJun 27, 2026, 2:22 AM
githubUrlhttps://github.com/neomjs/neo/issues/14146
authorneo-opus-grace
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 27, 2026, 2:22 AM

P0: KB-sync shadow-swap discards ALL embedding progress on one failed batch — no checkpoint/resume

Closed v13.1.0/archive-v13-1-0-chunk-7 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Jun 26, 2026, 11:26 PM

Operator-reported P0 (2026-06-26). A KB sync failed at embedding batch 214 of 328 (Failed to process batch 214 after 5 retries. Aborting.) and lost all 213 completed batches — the re-run restarts from batch 1, re-embedding all 16382 chunks. A ~30-second provider blip cost the entire corpus re-embed.

Root cause of the data-loss (Neo-side, independent of what tripped batch 214)

VectorService.embedViaShadowSwap is all-or-nothing. It builds the entire shadow collection across every batch and only promotes on full success. A single failed batch — embedChunks throw at VectorService.mjs:716 — is caught at line 744, the incomplete shadow is parked/discarded (parkFailedShadowCollection), and the next run rebuilds from scratch. So any single transient embedding-batch failure discards all completed embedding progress, plus leaves a window with no refreshed KB.

This is the amplifier that turns a transient provider hiccup into a P0: the trigger (one batch's The requested resource could not be found) is recoverable-in-principle, but the swap design converts it into total-progress-loss.

Fix direction (one or more)

  1. Checkpoint / resume — persist embedded-batch progress so a re-run resumes from the last completed batch. The shadow collection already holds the completed batches at failure time; resume into it instead of discarding + rebuilding from batch 1.
  2. Resilient batch handling — a bounded transient batch failure should retry/skip-and-record without aborting the whole shadow-swap; abort only on a systemic/persistent failure (mirrors the v13.1 systemic-fault-bound discipline).
  3. Recognize provider 404 in the retry guardTextEmbeddingService.#postOpenAiCompatible's isModelLoadError only matches HTTP-400 Model was unloaded / Failed to load model shapes; an HTTP-404 gets 5 blind re-POSTs then aborts. Sub-fix (the trigger's true cause is provider-side per the incident).

Acceptance

  • A single transient embedding-batch failure does NOT discard completed embedding progress — the sync resumes from / retains the completed batches.
  • (Optional sub) the retry guard recognizes recoverable provider 404s rather than blind-retrying then aborting.

Related

  • Trigger was a provider 404. Config-hygiene cleanup (separate): openAiCompatible.host defaults to http://127.0.0.1:11434 (Ollama's port) while LM Studio runs on :1234 (= orchestrator.lms.port) — the tracked default points at the wrong port for an LMS deployment.
  • Part of #14039 (v13.1 Agent OS Stability / data-integrity immune system) — this is the KB-sync analogue of the Memory Core data-recovery work.

Authored by Grace (Claude Opus 4.8, Claude Code).

tobiu referenced in commit 61634f3 - "feat(ai): KB shadow-swap resume — preserve on transient failure, resume from completed batches (#14146) (#14161) on Jun 27, 2026, 2:22 AM
tobiu closed this issue on Jun 27, 2026, 2:22 AM