Embedding admission has no aging: interactive canaries can defer batch ingestion indefinitely

Context

Live external-plane evidence, 2026-08-13 19:59Z, read through the orchestrator deployment-state bridge (inspect_deployment). The plane has four configured tenant repos. Exactly one has ever ingested. The other three carry lastIngestedRev: null with 40, 56 and 62 consecutive failures and sit at the 2-hour backoff cap. Credentials and config are clean — accessReadiness: ready 4/4, degraded 0, config: repoCount 4, disabledCount 0, errors [] — so the failure is entirely at the embed step.

The orchestrator log shows the work arriving correctly and dying at the last moment:

apps-ide   materialized: envelopeFiles=1586 ingested=2260 embeddings=0 errors=1
apps-ide   deferred: embedding incomplete, checkpoint held at none
           codes=KB_VECTOR_EMBED_CONNECTION_REFUSED ingested=2260 embeddings=0 (streak held at 62)
apps-suite ingested=197 embeddings=0

2,260 items materialized from one repo, zero embedded, checkpoint held at none, nothing committed. This has been the steady state for weeks on that plane.

The Problem

providerActivity aggregates from the same snapshot, one engine, same model (qwen3-embedding-8b, openAiCompatible):

Two independent health canaries — one per server — dispatch at interactive priority against the same engine that tenant ingestion must share. They are admitted essentially instantly. Real ingestion is batch and waits 2.8 minutes on average and up to 17.5 minutes before it starts.

What the ledger proves — and what it does not (corrected 2026-08-14; the original paragraph here is preserved in the correction comment). The waits are real and unbounded: batch work can be overtaken indefinitely with no aging, which is a forward-progress defect on its own. But the original claim — that queue wait pushed ingestion past "the 300s deadline" — was wrong twice over. The observed plane's batch-embed ceiling is 3,600,000 ms; 300 s is the healthcheck probe's deadline, a different control loop. And the failures quoted above are KB_VECTOR_EMBED_CONNECTION_REFUSED — produced by the container restart storm (#17063 / #17065) killing in-flight work, not by queue-induced timeouts. The starvation documented here is an amplifier in that incident, not its root cause. It still needs fixing on its own terms: a scheduler whose "lower priority" can degenerate into "never" is broken under any continuously-polling higher-priority producer. And the 12.8 s average execution of tiny canary inputs — itself a symptom of engine thread oversubscription inside the CPU quota — is what made the instrument's call volume expensive enough to matter.

The health probe consumes the capacity it exists to measure. 506 canary calls against 169 ingestion calls: the instrument is three times the volume of the work. Both canaries poll continuously and independently, neither is aware the other exists, and neither yields to batch work that has already been waiting minutes.

Each failed ingestion increments the per-repo failure counter, which drives backoffCapMs to its 2-hour ceiling — so the lane is suppressed for hours after each preemption, and a repo can go weeks without a single successful sweep.

Relationship to existing tickets — this is the opposite direction

#17048 documents the same engine and same plane in the other direction: one wide multi-input batch fans out across every n_parallel slot and starves the canaries queued behind it. Its fix caps batch width and separates per-class deadlines.

This ticket is the inverse and is not covered by that work: nothing protects batch work from interactive work. #17048 leaves a slot interleavable for canaries; no mechanism bounds how much of the engine the canaries themselves consume, nor prevents a batch item from being overtaken indefinitely. Both starvations are real on the same plane on the same day, which is why fixing only #17048 would not have moved this number.

Adjacent but distinct: #16972 (timed-out batch retried at identical size — compounds this by re-buying the same doomed request); #16853 (early abort strands provider work — explains why abandoned attempts keep burning engine capacity).

Architectural Reality

• ai/services/shared/providerActivityLedger.mjs is observer-only telemetry (corrected 2026-08-14 — the original bullet claimed its nativeAdmission accounting drives ordering; it does not). Admission authority is process-local: each of the KB server, MC server, and tenant-ingestion orchestrator owns a private OpenAI-compatible queue inside its TextEmbeddingService, and the selector #getNextOpenAiCompatiblePostQueueIndex() is the only ordering decision point.
• The canary is emitted by the KB and MC health surfaces; it is a liveness probe, so its result is only meaningful as "can the provider answer at all".
• Ingestion dispatch is priority: batch by correct intent — it is bulk work — but priority here decides admission order with no aging, so "lower priority" degenerates into "indefinitely deferred" whenever a higher-priority producer polls on a fixed cadence.

The Fix (shape, not prescription)

(Items 2 and 3 struck 2026-08-14 — see the Acceptance Criteria dispositions below; the original items are preserved in the correction comment.)

1. Age batch work into contention. A queued item's effective priority must rise with wait time, so a batch item that has waited past a bound cannot be overtaken again by a fresh interactive arrival. Starvation-freedom is the property; the exact ordering policy is open.

Acceptance Criteria

(Narrowed 2026-08-14 after @neo-gpt's source-cited pre-execution challenge below. The original five ACs are preserved verbatim in the correction comment, each with the source coordinate that settled it.)

• A queued batch embedding item is overtaken by at most ONE interactive selection within its owning process-local queue; when both lanes remain queued, the oldest batch wins the next selection, FIFO stays intact within each lane, and the aging state dies with the task so an aborted batch cannot bias a successor. A mutation-sensitive fixture proves dispatch order blocker → interactive-1 → batch → interactive-2 while preserving interactive-first for the first overtake.

Struck ACs — dispositions (2026-08-14):

• Canary-volume bounding (was AC2): deferred-to-clean-measurement. The instrument's cost term (12.8s per tiny embed) was thread-oversubscription overhead, removed by #17073 / PR #17074. Whether residual canary volume is still material is answerable only by the post-deploy clean read that #16706's S1/S2 verification owns; if it is, a dedicated ticket gets filed with clean numbers instead of speculation from pathology-era data.
• Queue-wait reason code (was AC3): already met in source. The batch HTTP timeout starts after local queue admission — #drainOpenAiCompatiblePostQueue dequeues before #postOpenAiCompatible runs — so queue wait is never charged to the provider deadline, and failureStage: 'queue' already distinguishes queued aborts from provider failures (ai/services/memory-core/TextEmbeddingService.mjs).
• Backoff not incremented on preemption (was AC4): already met in source. Embedding-incomplete sweeps hold the failure streak — this ticket's own quoted orchestrator log says streak held at 62.
• Measured-shape replay (was AC5): narrowed into the AC above. The 4-slot / 15s-cadence replay presumed one shared cross-process queue; scheduling authority is process-local, so the starvation property is proven at the queue owner, where it lives.

Out of Scope

• Batch-width capping and per-class deadlines (#17048 owns those).
• Retry-size adaptation (#16972).
• Provider-side stranded runners (#16853).
• Any change to embedding model choice, engine tuning, or host resource allocation — the ordering defect stands independently of capacity (see Avoided Traps for the corrected framing).

Avoided Traps

• "Give the engine more CPU." Insufficient rather than wrong (corrected 2026-08-14): on the observed plane most "load" was thread-oversubscription overhead, and removing that overhead shrinks every number in the table — but no amount of capacity makes an admission policy without aging starvation-free. The ordering defect stands independently of capacity.
• "The provider is at fault." The error codes say provider failure; the ledger says the request waited 2.8 minutes before it was allowed to start. Treating the symptom as a provider problem is what kept this open.
• "#17048 covers it." It covers the mirror image. Both directions are live on the same plane.

Related

• #17048 — engine-slot monopoly, opposite direction, same plane and day
• #16972 — identical-size retry after timeout
• #16853 — early abort strands provider work