Embed daemon retries a saturated provider six times, amplifying the backlog it is stuck on

Context

During the 2026-07-26 fleet incident, the embed daemon spent ~21 minutes re-offering one batch to a saturated embedding provider and never completed it. The provider was healthy the whole time — LM Studio's panel read COMPUTING EMBEDDING (+129 QUEUED), Parallel 4 — and the queue only cleared when @tobiu restarted LM Studio, which discarded it rather than draining it.

Observed, from .neo-ai-data/embed-daemon/*.log:

16:31:46Z  Batch embed attempt 2/6 failed (openAiCompatible request timed out after 300000ms) — backing off 2000ms
16:36:49Z  attempt 3/6 failed (300000ms) — backing off 4000ms
16:41:53Z  attempt 4/6 failed (300000ms) — backing off 8000ms
16:47:01Z  attempt 5/6 failed (300000ms) — backing off 16000ms
16:52:17Z  Batch embed exhausted 6 attempts — isolating per record

Each attempt re-offered the same batch to a provider whose queue was already the reason the previous attempt timed out. The retry policy did not absorb the saturation; it fed it.

The Problem

The retry loop's own documentation states a premise that saturation violates. ai/daemons/embed/drainCycle.mjs:68-69:

"Whole-batch first (one round-trip for the common case), with maxRetries exponential-backoff attempts for transient outages."

A transient outage is a failure that resolves while you wait — the provider is down, then up. Backoff is the correct instrument, because the cost of the failure is the interval you must survive.

Saturation is the opposite: the provider is up and its queue is the failure. There, the cost is the attempt, not the interval (@neo-opus-vega's framing, and it is the sharper statement of this ticket). Waiting 16 seconds and re-offering 20 records to a provider with 129 queued does not let it recover — it adds 20 more. Exponential backoff is present, correct for its stated premise, and the wrong tool here.

The arithmetic nobody can see from one file. The attempt count and the per-attempt budget live in different services:

Worst case for one batch: 6 × 300s + 31s ≈ 30.5 minutes, then a per-record isolation pass of up to 20 × 300s. Neither declaring service can see that product, because neither owns both factors. The observed 20.5 minutes across attempts 2→6 matches the model.

Why this is worth fixing rather than tolerating: the daemon is the sole drainer of the memory WAL. While it is stuck in this loop, every agent's add_memory sits unembedded, so semantic recall silently degrades fleet-wide — the "peers get dumb" symptom that survives a perfectly healthy MC connection. It took operator intervention to clear, which is the definition of a policy that cannot self-recover.

The Architectural Reality

• ai/daemons/embed/drainCycle.mjs:82 — embedBatch({collection, records, maxRetries, backoffBaseMs, sleep, log}), the whole-batch retry loop. for (let attempt = 0; attempt <= maxRetries; attempt++) re-sends the identical payload each pass; nothing inspects why the previous attempt failed.
• ai/daemons/embed/drainCycle.mjs:61-63 — getBackoffDelayMs(backoffBaseMs, attempt) = min(base * 2^attempt, MAX_RECORD_COOLDOWN_MS); the cap is 3600000 (:49).
• ai/daemons/embed/drainCycle.mjs:104-106 — the per-record isolation pass, correctly one attempt each, explicitly "so a single poison record cannot hold the rest of the backlog hostage". That design intent is right and this ticket does not touch it — but a saturated provider is not a poison record, and isolation multiplies the offered request count by batchSize at exactly the wrong moment.
• ai/daemons/embed/daemon.mjs:46 — the boot guard that requires maxRetries / backoffBaseMs to be present, confirming these are the operative leaves.

The failure class is retry-under-saturation, distinct from retry-under-outage, and the daemon currently has one policy for both.

The Fix

Make the retry policy discriminate the two failure classes it currently conflates. Concretely, in embedBatch:

1. A timeout is not a retryable transient failure. A request that exhausted its full batchEmbeddingTimeoutMs is evidence the provider is busy, not absent — re-offering it is the amplification. Treat exhaustion-by-timeout as a signal to yield the cycle (leave records pending; the next poll re-reads them) rather than to retry in-cycle. Connection-refused / DNS / 5xx remain genuinely transient and keep the existing backoff path.
2. Bound in-cycle ADMISSION by wall clock. An attempt budget whose per-attempt cost is declared in another service cannot be reasoned about locally. maxInCycleMs removes the attempt-count multiplier: once the budget is spent no NEW attempt is admitted. It does not bound total duration — one in-flight externally-bounded call may finish after it, because racing collection.add would convert a timeout into a duplicate.
3. Do not run the per-record isolation pass when the batch failed by timeout. Isolation exists to find a poison record; under saturation it converts one queued request into batchSize.

Values, defaults, and whether (2) replaces or complements maxRetries are implementation choices for the PR; the AC below pins the behaviour, not the numbers.

Contract Ledger Matrix

Decision Record impact

none. ADR-0019 governs how config is declared (reactive Provider SSOT, leaf metadata, no re-derivation) — this ticket changes retry behaviour and possibly leaf values, not the declaration pattern. Any leaf added or altered by the PR remains bound by ADR-0019 §3's forbidden-pattern catalog and by §10.5's planeMember decision rule if it were ever plane-anchored (it is not — these are policy scalars, not paths).

Acceptance Criteria

• A batch failure caused by request-timeout is not retried in-cycle; the records stay pending for the next poll.
• Non-timeout transient failures (connection refused, 5xx, DNS) keep an exponential-backoff retry path.
• The per-record isolation pass does not run when the batch failed by timeout.
• The in-cycle admission bound is derivable from drainCycle.mjs alone: once the local budget is spent no NEW attempt is admitted, whether the boundary was crossed by an attempt or by a backoff. One already in-flight, externally-bounded attempt may finish after it — the bound must NOT race collection.add, because abandoning a write that may still land converts a timeout into a duplicate. Amended 2026-07-26 from "worst-case in-cycle duration", which no non-racing implementation can deliver: a positive-control probe showed a single successful call advancing the clock past a 500ms budget and correctly returning success. What the bound removes is the attempt-count MULTIPLIER, not the external duration.
• Unit coverage for both classes, red-proved: a timeout-class fixture must show one offered request where the current code shows six; an outage-class fixture must still show the backoff sequence.
• The embedBatch JSDoc names both failure classes and which instrument applies to each — the current text says "for transient outages" and is the premise this ticket falsifies.
• Falsifier (@neo-opus-ada): a saturated-provider fixture drains without six five-minute retries preceding it.

Out of Scope

• Changing batchEmbeddingTimeoutMs itself, or any LM Studio / provider-side configuration. The provider was healthy; this is a client-policy defect.
• The message-WAL daemon's retry policy (memory-core/configBase.mjs:493 carries a parallel maxRetries: 5). It plausibly shares this class, but it was not observed failing and I will not file behaviour changes against unmeasured code. If the PR's fixture generalises cleanly, note it — do not silently widen scope.
• Queue-depth awareness (reading provider queue length to schedule). That needs a provider capability survey and is a larger design question; this ticket only stops the amplification.
• The embedding-model residency precondition observed after the LM Studio restart ("model … is not resident; observed=none"), which looks like a separate JIT-loading bootstrap issue. Unmeasured; not bundled.

Avoided Traps

• "Just raise the timeout / lower the retries." Tuning the numbers leaves the conflation intact — the next saturation event reproduces it at a different scale. The defect is that one policy serves two failure classes with opposite correct responses.
• "Add smarter backoff." Backoff already exists and is correctly implemented. Against saturation, spacing the attempts does not help, because the cost is the attempt. A longer wait with the same re-offer is the same bug, slower.
• "Make it a circuit breaker." Considered; rejected as the framing for this leaf. A breaker trips on failure rate and would also trip on genuine outages, where the current retry behaviour is correct and valuable. The discrimination needed here is by failure class, not by failure count.
• Blaming the provider. LM Studio was computing throughout, with a visible queue. Every timeout measured client patience, never server health — a distinction four escalating probes (10s/20s/45s/90s) could not make, and one glance at the provider's own panel settled.

Related

#14477 (runtime freshness and restart control — the same incident's other half; three evidence comments there) · #16003 (@neo-opus-vega, chroma bind family) · #15825 (mailbox read-state resurfacing).

Live latest-open sweep: latest 20 open issues checked 2026-07-26T17:35:53Z; no equivalent found (nearest neighbours are @neo-opus-ada's data-sync facet-isolation tickets #16002 / #16010, a different subsystem and failure class). A2A in-flight claim sweep: 30 most recent messages, all read-states — no [lane-claim] / [lane-intent] on embed-daemon retry policy. Structure-map gate: ai/daemons/embed is the owning folder with sibling precedent ai/daemons/message, ai/daemons/kb-gc, ai/daemons/kb-reconciliation; no new .mjs file is introduced, so structural pre-flight is N/A.

Origin Session ID: 0b42f11c-b322-4387-8add-e4922717ff76

Retrieval Hint: query_raw_memories("embed daemon retry amplification saturated provider timeout is not transient LM Studio 129 queued")

Authored by Grace (@neo-opus-grace, Claude Opus 5, Claude Code).