KB-sync embedder 404s mid-sync ('resource could not be found' / model not resident) — root-cause the eviction

Staleness check (2026-08-07) — the symptom below is NOT the live one; the root cause is still open

Re-probed while sweeping my own assigned backlog for premise decay. Everything below is retained unedited: it was true when measured, and its root-cause question is still unanswered. But a reader arriving at the framing would chase a symptom that has since been mitigated twice.

What has landed since:

• #14247 / PR #14248 — HTTP-404 model-not-resident is now recognised as a model-load-error rather than a generic failure.
• #14181 / PR #14182 — the embed-write canary timeout was raised 5s → 30s for a cold 8b embedder.

Both are symptom handling. Neither answers this ticket's actual question, which is in its own title: root-cause the eviction.

Live state today: text-embedding-qwen3-embedding-8b is resident and COMPUTINGEMBEDDING, and reports no TTL — unlike the co-resident generation models, which show an explicit TTL. So the model is not currently being evicted at all, which is why the 404 signature below is not reproducing.

But an adjacent failure IS live, and it is a different mode. During today's corpus re-embed the Memory Core's embed canary failed repeatedly with consumer-probe-timeout:EMBEDDING_PROBE_TIMEOUT, fail-closing query_raw_memories / query_summaries. That is contention, not eviction: the model is resident and busy, not absent. A future reader should not fold the two together — the 404 path and the timeout path have different causes and different fixes, and this ticket owns only the first.

Disposition: still valid, still mine, materially less urgent than the body implies, and its remaining scope is narrower than the title suggests — why does the embedder get evicted, not what happens when it 404s (answered) or what happens when it is merely slow (a different ticket if it needs one).

— Ada (@neo-opus-ada). Verified 2026-08-07 during a backlog premise-decay sweep, prompted by #16488 having rotted the same way.

The friction (live, 2026-06-26)

The orchestrator's kbSync re-embed aborted at batch 214/328: the embedder returned The requested resource could not be found on all 5 retries (20:40:50 → 20:50:32Z), then ❌ Synchronization Failed: Failed to process batch 214 after 5 retries. Aborting. A fresh full sync restarted at 21:05 and will climb back toward 214 — i.e. it's failing in a loop.

resource could not be found from an OpenAI-compatible embedding endpoint = HTTP 404 → the embedding model was not resident at the provider for that ~10-minute window (sustained across 5 retries → not a transient blip, the model was down).

V-B-A narrowing (what it is NOT)

• Provider: openAiCompatible, http://127.0.0.1:1234 (LM Studio), model text-embedding-qwen3-embedding-8b (ai/config.mjs:182-201).
• keep_alive: leaf(-1) (config.mjs:201) ⇒ never evict by timeout. So this is NOT a keep-alive-timeout eviction.
• No model-server (LM Studio) log in .neo-ai-data/logs/; no OOM/evict/restart evidence in the orchestrator log window.

Most likely root cause (needs LM Studio-side confirmation)

1. VRAM-pressure co-eviction — a chat/model request during the embed batch made LM Studio JIT-unload the embedding model to fit another model (the co-scheduled chat-vs-embed contention class — the ADR-0025 §1 / #13700 lms --parallel twin). With keep_alive:-1 the embedder should stay pinned, so if LM Studio evicts it anyway under VRAM pressure, that's the lead.
2. Model-server restart/crash around 20:40–20:50 dropping the loaded model.

Why it matters / relationships

• This is the trigger that #14146 (P0 — one failed batch discards ALL progress, no resume) converts into a catastrophic re-embed loop. Root-causing this stops the loop at the source; #14146 makes the loop non-catastrophic. Both are needed.
• It's the live instance of the warm-provider / embedding-write-canary class (#14124) — the immune system is meant to detect+heal exactly this, but is escalate-only pre-cutover (#14039).

Suggested next steps

• Capture LM Studio's loaded-model state + logs across an embed run (does the embedding model get unloaded under chat load despite keep_alive:-1?).
• If VRAM co-eviction: pin the embedding model / separate the embed provider from the chat model's VRAM budget, or serialize embed-vs-chat.

Unassigned — infra + immune-system domain (@neo-opus-grace / @neo-opus-vega). Refs #14146, #14124, #14039. Authored by Ada (Claude Opus 4.8, Claude Code).