Context
An over-cap embedding input (a too-large source/document before chunking existed) was observed to freeze the embedding provider — pegging multiple ollama worker CPU cores at 100% with no completion — which both stalled the deferred-embed WAL drain (a metadata-without-vector pathway, see #14027/#13999) and burned the host. The pre-call mitigations since added (input guards #13928, context-truncation #13944, oversized chunking #14000) prevent the known over-cap cause, but there is no signal that catches a frozen/runaway embed call from any cause while it is happening.
The Problem
Provider readiness checks (#13851 loaded-context, #13950 load-aware, residency diagnostics) verify the model is up before a call. Input guards reject known-oversized inputs before a call. Neither detects a call that has started and is hung/runaway (over-cap that slipped a guard, a provider bug, a pathological input). A single hung embed call can stall the WAL drain indefinitely and saturate CPU — silently, until a downstream symptom (failed backup, coverage drop) surfaces days later.
The Architectural Reality
- Shared embedding path:
TextEmbeddingService.embedTexts (used by KB + MC); the embed-daemon drains the add_memory WAL through it (#12838/#12840).
- #13551 (embed-drain watchdog) alarms on a stalled WAL backlog — a downstream/aggregate signal; this ticket is the per-call upstream signal that explains why the drain stalled.
- ADR-0025 diagnostics detect container CPU saturation, but not per-embed-call duration.
The Fix
A per-embed-call watchdog: bound each embed call with a duration timeout + emit a signal when a call exceeds a threshold (and/or when embedding-provider CPU saturates with no throughput). On timeout: abort the call, route the input to the unrecoverable/retry path (fail-loud), and emit an ADR-0025 diagnosis. Prevents one hung call from stalling the whole drain.
Acceptance Criteria
Out of Scope
- Pre-call input guards / chunking (#13928 / #14000 — complementary, already landed).
- The WAL-drain backlog watchdog (#13551 — the downstream/aggregate complement).
Related
- #14027 / #13999 — the freeze→WAL-drain-stall→metadata-without-vector mechanism this catches at the source.
- #13551 — embed-drain backlog watchdog (downstream complement).
- #14026 — data-integrity detect-signal (sibling).
- #13928 / #13944 / #14000 — pre-call over-cap mitigations.
Origin Session ID: 58acd7a6-778e-4fe6-b85a-4569802ac57b
Handoff Retrieval Hints: query_raw_memories("embedding provider freeze runaway per-call duration timeout WAL drain stall CPU saturation"); anchors: TextEmbeddingService.embedTexts, embed-daemon WAL drain (#12838/#12840), #13551.
Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8)
Context
An over-cap embedding input (a too-large source/document before chunking existed) was observed to freeze the embedding provider — pegging multiple ollama worker CPU cores at 100% with no completion — which both stalled the deferred-embed WAL drain (a metadata-without-vector pathway, see #14027/#13999) and burned the host. The pre-call mitigations since added (input guards #13928, context-truncation #13944, oversized chunking #14000) prevent the known over-cap cause, but there is no signal that catches a frozen/runaway embed call from any cause while it is happening.
The Problem
Provider readiness checks (#13851 loaded-context, #13950 load-aware, residency diagnostics) verify the model is up before a call. Input guards reject known-oversized inputs before a call. Neither detects a call that has started and is hung/runaway (over-cap that slipped a guard, a provider bug, a pathological input). A single hung embed call can stall the WAL drain indefinitely and saturate CPU — silently, until a downstream symptom (failed backup, coverage drop) surfaces days later.
The Architectural Reality
TextEmbeddingService.embedTexts(used by KB + MC); the embed-daemon drains the add_memory WAL through it (#12838/#12840).The Fix
A per-embed-call watchdog: bound each embed call with a duration timeout + emit a signal when a call exceeds a threshold (and/or when embedding-provider CPU saturates with no throughput). On timeout: abort the call, route the input to the unrecoverable/retry path (fail-loud), and emit an ADR-0025 diagnosis. Prevents one hung call from stalling the whole drain.
Acceptance Criteria
Out of Scope
Related
Origin Session ID: 58acd7a6-778e-4fe6-b85a-4569802ac57b
Handoff Retrieval Hints:
query_raw_memories("embedding provider freeze runaway per-call duration timeout WAL drain stall CPU saturation"); anchors:TextEmbeddingService.embedTexts, embed-daemon WAL drain (#12838/#12840),#13551.Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8)