Health canaries outrank tenant ingestion in the embedding admission queue, so ingestion never finishes inside its own deadline

Context

Live external-plane evidence, 2026-08-13 19:59Z, read through the orchestrator deployment-state bridge (inspect_deployment). The plane has four configured tenant repos. Exactly one has ever ingested. The other three carry lastIngestedRev: null with 40, 56 and 62 consecutive failures and sit at the 2-hour backoff cap. Credentials and config are clean — accessReadiness: ready 4/4, degraded 0, config: repoCount 4, disabledCount 0, errors [] — so the failure is entirely at the embed step.

The orchestrator log shows the work arriving correctly and dying at the last moment:

apps-ide   materialized: envelopeFiles=1586 ingested=2260 embeddings=0 errors=1
apps-ide   deferred: embedding incomplete, checkpoint held at none
           codes=KB_VECTOR_EMBED_CONNECTION_REFUSED ingested=2260 embeddings=0 (streak held at 62)
apps-suite ingested=197 embeddings=0

2,260 items materialized from one repo, zero embedded, checkpoint held at none, nothing committed. This has been the steady state for weeks on that plane.

The Problem

providerActivity aggregates from the same snapshot, one engine, same model (qwen3-embedding-8b, openAiCompatible):

Two independent health canaries — one per server — dispatch at interactive priority against the same engine that tenant ingestion must share. They are admitted essentially instantly. Real ingestion is batch and waits 2.8 minutes on average and up to 17.5 minutes before it starts.

The arithmetic is the defect. Ingestion's own execution (183s average) fits inside the 300s deadline. Queue wait (167s) plus execution (183s) is 350s, which does not. Ingestion therefore times out on a deadline it would otherwise meet, and the timeout is attributed to the provider (KB_VECTOR_EMBED_FAILED / KB_VECTOR_EMBED_CONNECTION_REFUSED) rather than to the queue that caused it.

The health probe consumes the capacity it exists to measure. 506 canary calls against 169 ingestion calls: the instrument is three times the volume of the work. Both canaries poll continuously and independently, neither is aware the other exists, and neither yields to batch work that has already been waiting minutes.

Each failed ingestion increments the per-repo failure counter, which drives backoffCapMs to its 2-hour ceiling — so the lane is suppressed for hours after each preemption, and a repo can go weeks without a single successful sweep.

Relationship to existing tickets — this is the opposite direction

#17048 documents the same engine and same plane in the other direction: one wide multi-input batch fans out across every n_parallel slot and starves the canaries queued behind it. Its fix caps batch width and separates per-class deadlines.

This ticket is the inverse and is not covered by that work: nothing protects batch work from interactive work. #17048 leaves a slot interleavable for canaries; no mechanism bounds how much of the engine the canaries themselves consume, nor prevents a batch item from being overtaken indefinitely. Both starvations are real on the same plane on the same day, which is why fixing only #17048 would not have moved this number.

Adjacent but distinct: #16972 (timed-out batch retried at identical size — compounds this by re-buying the same doomed request); #16853 (early abort strands provider work — explains why abandoned attempts keep burning engine capacity).

Architectural Reality

• ai/services/shared/providerActivityLedger.mjs owns the activity records and the nativeAdmission accounting the ordering reads.
• The canary is emitted by the KB and MC health surfaces; it is a liveness probe, so its result is only meaningful as "can the provider answer at all".
• Ingestion dispatch is priority: batch by correct intent — it is bulk work — but priority here decides admission order with no aging, so "lower priority" degenerates into "indefinitely deferred" whenever a higher-priority producer polls on a fixed cadence.

The Fix (shape, not prescription)

1. Age batch work into contention. A queued item's effective priority must rise with wait time, so a batch item that has waited past a bound cannot be overtaken again by a fresh interactive arrival. Starvation-freedom is the property; the exact ordering policy is open.
2. Bound canary cost against the lane it measures. Two servers independently polling one engine is the amplifier. Options: share one canary result across KB and MC within a freshness window; skip the canary entirely while a known batch is in flight and report busy-behind-batch (the class #17048 already proposes); or cap canary share of engine capacity.
3. Do not charge queue wait to the provider deadline — or, if the deadline is end-to-end by design, report the timeout with its queue-wait component so a queue-induced failure is never recorded as a provider fault. Today the ledger has the evidence and the error code discards it.

Acceptance Criteria

• A batch embedding item that has waited past a defined bound is admitted ahead of a newly-arriving interactive canary; a fixture with a continuously-polling canary and one batch item proves the batch completes.
• With both KB and MC canaries active against a single-engine fixture, total canary calls over a window are bounded relative to ingestion calls (today: 506 vs 169).
• An embed failure whose queue wait exceeded its execution time is reported with a distinguishing reason code, not as a bare provider failure.
• A repo whose sweep fails only because of queue preemption does not increment the consecutive-failure counter that drives the 2-hour backoff, OR the backoff is cleared when the cause is reclassified.
• Regression: replay the measured shape — 4-slot engine, two interactive pollers at ~15s cadence, one batch lane — and assert the batch lane makes forward progress.

Out of Scope

• Batch-width capping and per-class deadlines (#17048 owns those).
• Retry-size adaptation (#16972).
• Provider-side stranded runners (#16853).
• Any change to embedding model choice, engine tuning, or host resource allocation — this is an ordering defect and reproduces regardless of how much capacity the engine has.

Avoided Traps

• "Give the engine more CPU." Rejected. The lane is not compute-bound; it is order-bound. More capacity would be consumed in the same 3:1 ratio and would move the threshold without removing the starvation.
• "The provider is at fault." The error codes say provider failure; the ledger says the request waited 2.8 minutes before it was allowed to start. Treating the symptom as a provider problem is what kept this open.
• "#17048 covers it." It covers the mirror image. Both directions are live on the same plane.

Related

• #17048 — engine-slot monopoly, opposite direction, same plane and day
• #16972 — identical-size retry after timeout
• #16853 — early abort strands provider work
• #17044 — provider-lane health signal family