Parallelize Memory Core re-embed per-document isolation fallback

Context

Surfaced by @tobiu while watching the live #13999 Memory Core recovery on 2026-06-26: the neo-native-graph collection re-embedded its missing-vector rows one-at-a-time, while neo-agent-memory / neo-agent-sessions had embedded in batches. The local embedding provider can serve multiple concurrent requests (parallel), so the sequential leg leaves provider bandwidth on the floor and makes the recovery's failure path several times slower than it needs to be.

V-B-A (read-only, against dev): the divergence is not collection-specific config — it is a failure-path fallback that triggered for graph because its larger documents made the batch embed call throw.

The Problem

extractMemoryCoreCollectionData slices missing-vector rows into batches and re-embeds each batch through embedFn (= TextEmbeddingService.embedTexts bound to the MC provider). When a whole-batch embed succeeds, throughput is fine. When it throws (e.g. one oversized graph-node document, or a payload that exceeds the embedding context), it drops into a per-document isolation fallback that re-embeds every document sequentially to pin down which one is unrecoverable.

That isolation is correct — it must attribute the failure to the exact row — but running it as a serial await loop discards all provider concurrency. Empirical cost on the 2026-06-26 recovery: neo-agent-sessions re-embedded 886 rows in ~12 min (batched), while neo-native-graph took >21 min for 309 rows in the sequential fallback — roughly 4-5× slower per row. For a self-healing recovery actuator (v13.1), recovery wall-time is a real property: slower recovery = longer lease-hold and longer operator wait.

The Architectural Reality

• ai/scripts/maintenance/repairMemoryCoreStoredEmbeddings.mjs:270-302 — embedRecoverableDocuments. Happy path (:272) embeds the whole batch via embedFn(documents). The catch (:279-299) is a sequential for loop: await embedFn([documents[i]]) per document, collecting failures with {index, reason, message}.
• ai/services/memory-core/TextEmbeddingService.mjs:674-712 — #embedOpenAiCompatibleTexts chunks texts into batchEmbeddingChunkSize (default 5, :688) and processes the chunks sequentially with a deliberate yield between them (:708-709), backed by an interactive-vs-batch priority queue (:161, :217). This serialization is by design: it lets latency-sensitive add_memory single-embeds preempt a multi-minute batch instead of starving behind it.

The key distinction: the batch-chunk serialization in the service is a deliberate interactive-fairness mechanism; the isolation-fallback serialization in the repair script has no such rationale — it is incidental, and during a recovery there is no interactive traffic to protect.

The Fix

Make the per-document isolation fallback issue its single-document embeds with bounded concurrency (cap = provider embedding parallelism, configurable; conservative default), routed through the existing batch-priority queue in TextEmbeddingService so interactive add_memory preemption is preserved. Isolation semantics are unchanged: each document is still embedded individually, so a failure is still attributed to the exact index/id; only the scheduling changes from serial to bounded-parallel.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR-0026 (recovery actuator envelope). No ADR amended — this is a throughput refinement of an existing actuator path, not a contract or topology change.

Acceptance Criteria

• The per-document isolation fallback embeds with bounded concurrency (configurable cap; default respects provider parallelism), not a serial await loop.
• Per-document failure attribution (index / reason / id) is preserved exactly: a failing document is still isolated and the rest still recovered.
• The concurrent per-doc embeds route through the existing interactive/batch priority queue, so add_memory interactive single-embeds still preempt batch work (no single-embed latency regression).
• Unit coverage: a batch with exactly one failing document recovers the rest concurrently, attributes the failure to the correct id, and never exceeds the concurrency cap.

Out of Scope

• Parallelizing the deliberately-sequential batch-chunk path in #embedOpenAiCompatibleTexts — that is the interactive-fairness tradeoff and belongs in a Discussion, not here (see Avoided Traps).
• Token-budget-aware batch sizing to avoid the whole-batch failure in the first place (separate, complementary concern).
• The in-flight live recovery — already running; this change does not affect it.

Avoided Traps

• "Just turn on parallel=N everywhere." The batch-chunk loop in TextEmbeddingService serializes on purpose (:677-679, :708-709) so interactive add_memory embeds can jump the queue between chunks. Blindly parallelizing that path would regress interactive latency. This ticket scopes only the isolation fallback, which carries no fairness rationale — keeping the safe win separate from the design question.

Related

• #14062 (partial-promotion recovery — sibling recovery-actuator slice), #14068 (repair parking bound / abort-predicate retirement follow-up).
• #14039 (v13.1 Agent OS Stability epic) — related, not a sub: recovery already functions; this is a post-v13.1 throughput optimization, so it does not extend the closing release scope.
• #13692 / #13695 (the original change that routed Chroma batch embeds through embedTexts).
• #13999 (the Memory Core recovery whose live run surfaced this).

A companion design question — whether the batch path should fan out concurrently up to the provider's parallel capacity while preserving interactive priority — is a candidate for the Ideation Sandbox (throughput-vs-fairness tradeoff; within a rate-limited peer's ideation budget).

Handoff Retrieval Hints

• query_raw_memories("Memory Core re-embed per-document isolation fallback sequential parallel recovery")
• Exact anchors: embedRecoverableDocuments (repairMemoryCoreStoredEmbeddings.mjs), #embedOpenAiCompatibleTexts / batchEmbeddingChunkSize (TextEmbeddingService.mjs), the 2026-06-26 #13999 recovery run.

Authored by Vega (Claude Opus 4.8) — surfaced by @tobiu during the 2026-06-26 live #13999 Memory Core recovery; premise V-B-A'd against dev.