The embedding health probe cannot fail the way the embedding lane fails

Context

On a deployment running the openAiCompatible embedding lane, 2026-08-18: the engine container was being OOM-killed on every band-sized batch, had accumulated 36 restarts, and had left three repositories backoff-suppressed with a knowledge base frozen at 152 vectors.

Throughout, tenantRepoSync.embeddingRecoveryProbe reported:

{ "status": "healthy", "failureStreak": 0, "backoffMs": 0,
  "terminal": false, "stopReason": null, "errorCode": null }

checkedAt was 08:30:22Z. The sweep's lastErrorAt was 08:30:20Z — two seconds earlier. The probe declared the lane healthy while the lane was failing, and it was not a stale read.

The Problem

The container logs show why, in one incarnation:

srv  llama_server: model loaded
srv  llama_server: listening on http://0.0.0.0:8080
slot      release: id  3 | task 0 | stop processing: n_tokens = 10,  truncated = 0
slot      release: id  3 | task 2 | stop processing: n_tokens = 9,   truncated = 0
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
   <log ends — next line is a fresh incarnation's startup banner>

The same process embedded a 10-token and a 9-token input successfully, then died launching a real one. The probe's input is tiny, so it fits in the memory the engine has left; production inputs run to roughly 14,000 tokens, and peak memory for one non-causal embedding request scales with the square of its token count. Measured on this exact image and configuration: idle 7.69 GiB, and a single 13,980-token embed peaking at 24.14 GiB.

So the probe and the workload differ by more than three orders of magnitude in the resource that is actually exhausted. A probe that cannot fail the way its subject fails is not a control — it certifies a property nobody needed (can the HTTP plane answer at all) while the property under question (can this lane deliver an admitted input) goes unmeasured.

The cost is not cosmetic. embeddingRecoveryProbe is the signal the tenant sweep consults to decide whether the lane has recovered, and status: healthy / failureStreak: 0 is a green light to re-dispatch. Every re-dispatch killed the engine again. The probe was actively driving the loop it was supposed to break.

The Architectural Reality

• tenantRepoSync.embeddingRecoveryProbe in the orchestrator deployment-state bridge — fields status, failureStreak, backoffMs, nextAttemptAt, terminal, stopReason, errorClassification, errorCode.
• The probe's verdict feeds re-dispatch eligibility in ai/daemons/orchestrator/services/TenantRepoSyncService.mjs.
• Related but distinct: the engine's own container healthcheck in the deployment is documented as "liveness only", deliberately, and that is correct for a healthcheck. The recovery probe is not a liveness check — it is consumed as a readiness verdict for real work.

This is the same class as the memory-shape mismatch on the sizing side: a control whose shape does not match the subject's shape cannot bound the subject's behaviour.

The Fix

The probe must exercise the dimension that fails, without becoming expensive enough to cause the problem it detects:

1. Give the probe a representative input, sized from the lane's admitted ceiling rather than an arbitrary short string — the largest input the geometry admits is the case worth probing, and a fixed tiny string is the one case guaranteed not to discriminate.
2. Where a full-ceiling probe is too costly to run at cadence, probe at a declared fraction of the ceiling and report the fraction alongside the verdict, so healthy reads as healthy at 25% of admitted size rather than as an unqualified pass.
3. Treat provider death during the probe as a distinct outcome from probe failure — the first is evidence the lane cannot serve the workload, the second may be ambient.
4. Never let status: healthy coexist with a sweep error timestamped inside the same probe interval without at least surfacing the disagreement in stopReason or errorClassification. Transferred to #17501 (2026-08-21). The probe/sweep reconciliation is an independent defect with its own residual owner: it concerns the relationship between two observers, while items 1–3 concern the probe's own representativeness — one PR discharging 1–3 cannot also carry 4's reconciliation contract without conflating the two surfaces. PR #17490 resolves this ticket on items 1–3; #17501 owns item 4's shape end to end. (Recorded per @neo-gpt's PR #17490 round-2 RA-3: the deliberate transfer belongs in this body, not only in the successor's.)

Contract Ledger Matrix

Decision Record impact

none.

Acceptance Criteria

• The probe's input size derives from the lane's admitted ceiling, not a constant.
• The probe result names the size it actually exercised, so healthy is never unqualified.
• A lane that serves a 10-token input and dies on an admitted-size input does not report status: healthy with failureStreak: 0.
• Provider death during the probe is classified distinctly from probe failure.
• Red-proof: a stubbed provider with a token-count-dependent failure threshold must report healthy on main and correctly report unhealthy after the change. A stub that fails on all inputs would pass both ways and prove nothing — the threshold is the point.
• Probe cost at cadence is bounded and stated.

Out of Scope

• The container healthcheck, which is correctly liveness-only by design.
• Graduating the offending chunk — #17336.
• Batch sizing and adaptive stride — #16972.
• The recovery actuator's admitted actions — separate ticket.

Avoided Traps

• Do not simply run the probe more often. Frequency does not fix a probe measuring the wrong dimension; it only reaches the wrong verdict sooner.
• Do not probe at the full ceiling unconditionally. At this geometry one ceiling-sized request costs ~24 GiB and ~180 s; a cadence probe of that weight would itself starve the lane. Declaring the fraction is what preserves honesty without the cost.
• Do not infer readiness from the container healthcheck instead. It answers liveness from a dedicated thread specifically so it stays answerable under full compute load — by construction it cannot see this failure.

Related

• #17336 — the graduation gap that lets the killing input be re-offered; this probe is what greenlights the re-offer.
• #17044 — self-heal futility breaker; both concern loops that fail to recognise their own futility.
• #16997 — an openAiCompatible timeout bypassing the tenant-run circuit; adjacent probe/circuit interaction.

Live latest-open sweep: checked latest 20 open issues at 2026-08-18T09:29:25Z plus a six-term state=all title sweep; no existing ticket covers probe/subject shape mismatch for the embedding lane.

Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2

Retrieval Hint: query_raw_memories("embedding recovery probe healthy while batches die single-item probe shape mismatch")

Body retarget 2026-08-21 (author). Fix item 4 (probe/sweep disagreement surfacing) is transferred to #17501, which owns the two-observer reconciliation end to end; this ticket resolves on items 1–3 via PR #17490. Recorded per @neo-gpt's round-2 RA-3 on that PR — a deliberate transfer is authoritative only when the transferring body says so itself.