Memory Core re-embed: chunk oversized docs that exceed the embed context

Context

During the 2026-06-26 #13999 Memory Core recovery, exactly 2 rows were unrecoverable — not because their data was missing, but because their documents are too large to embed:

aa8ecd3f-2315-4457-9c28-a19fc5005d76  reason: embedding-provider-error
  "LM Studio embedding context too small ... (loaded=32768, inputEstimate=45538)"
issue-11187  inputEstimate=35502 > 32768

These documents materialize fine; they simply exceed the embedding model's context window, so embedFn rejects them and they fall out of recovery permanently (they are the residue that keeps the repair non-clean — see Related).

The Problem

Memory Core's re-embed path has no oversized-document handling. When a document exceeds the provider's context, the per-document isolation pins it as unrecoverable and moves on — that memory is lost. The Knowledge Base already guards this dimension (filterEmbeddingInputBudget / over-budget skip, VectorService), but: (a) Memory Core has no equivalent, and (b) even KB's behavior is skip (leaving content unindexed, #14000), not recover. For Memory Core recovery we want oversized docs to remain searchable, not silently dropped.

The Architectural Reality

• ai/scripts/maintenance/repairMemoryCoreStoredEmbeddings.mjs — embedRecoverableDocuments calls embedFn(documents); on failure it isolates per-doc and records embedding-provider-error / context-exceeded as unrecoverable. No truncation/chunking is attempted.
• ai/services/memory-core/TextEmbeddingService.mjs — #assertOpenAiCompatibleEmbeddingContext is what throws when inputEstimate > context; the contract fails loud rather than truncating.
• KB precedent: ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs filterEmbeddingInputBudget + VectorService.embedChunks over-budget handling — the budget-aware pattern to mirror (but adapted from skip to recover).

The Fix

Give Memory Core re-embed a budget-aware path so oversized documents stay recoverable. Recommended shape (implementer chooses, floor → better):

1. Floor — truncate-to-context: embed a context-bounded prefix of the document so the row gets a vector (searchable, slightly lossy). Eliminates the unrecoverable class immediately.
2. Better — chunk-and-aggregate: split the oversized document into context-fitting chunks, embed each, and store a representative/aggregate vector (or multiple sub-vectors) so semantic search still finds it with full fidelity.

Apply at the re-embed boundary first (recovery), and consider the live MC write path so oversized memories never become unrecoverable in the first place (ties to the atomic-write invariant — see Related).

Contract Ledger Matrix

Decision Record impact

aligned-with #14029 (atomic vector-write invariant — a recoverable oversized doc is preferable to a durable vectorless row) and ADR-0017 (unified Chroma topology). No ADR amended.

Acceptance Criteria

• A Memory Core document exceeding the embedding context is recovered (receives a vector via truncation or chunking), not marked unrecoverable.
• The chosen approach is documented (truncate vs chunk-and-aggregate) with its fidelity tradeoff.
• A genuinely-unembeddable case that no strategy can handle still degrades cleanly to the existing unrecoverable classification (no silent data loss without a manifest entry).
• Unit coverage for the oversized-document recovery path.

Out of Scope

• The repair success-semantics / accepted-loss terminal (sibling ticket — this ticket shrinks that residue toward zero).
• KB's separate over-budget skip behavior (#14000).
• Re-architecting the embedding provider or raising the model context.

Avoided Traps

• Skip-and-lose (the KB pattern). For Memory Core recovery, an oversized memory must stay searchable; silently skipping it (KB's #14000 gap) is the wrong default here.
• Unbounded chunk fan-out. Chunking must be bounded; a pathologically large doc should cap its chunk count and fall back, not explode the embed workload.

Related

• #13999 / the 2026-06-26 recovery (where aa8ecd3f / issue-11187 became unrecoverable), #14066 (partial-promotion — these are its residue), #14029 (atomic vector-write invariant), #14000 (KB over-budget skip — the precedent + its gap), #14039 (v13.1 stability epic — the "Prevent" layer).

Handoff Retrieval Hints

• query_raw_memories("Memory Core oversized document embedding context exceeded chunk truncate re-embed unrecoverable")
• Exact anchors: embedRecoverableDocuments (repairMemoryCoreStoredEmbeddings.mjs), #assertOpenAiCompatibleEmbeddingContext (TextEmbeddingService.mjs), filterEmbeddingInputBudget (KB precedent).

Authored by Vega (Claude Opus 4.8, Claude Code) — surfaced by @tobiu during the 2026-06-26 #13999 recovery. Session c94ea3b2-1ae8-48fd-8f34-1c54d90f5caa.