Embedding lane consolidation: one authority for parallelism and geometry, and the layers it lets us retire

Problem scope

The embedding lane is defined by two numbers: how many requests may be in flight, and how many tokens one slot admits. Each is currently declared in roughly a dozen places — config leaves, per-service deployment env, the declared lane-shape mirror, and the provider's own arguments — and no consumer derives any of them from another. The numbers are consistent on paper, which is why this has been hard to see.

Measured on a live external tenant deployment, read-only, 2026-08-20 (revision e1e0517d4e):

The provider slot log shows ~20 consecutive launch → release → launch transitions, never two tasks in flight, slots selected round-robin by LRU. The dispatch loop awaits each POST before issuing the next (TextEmbeddingService.mjs:1930), and the sweep's concurrency is not a settable leaf (#17158).

Consequences, all measured rather than inferred:

• ~3.50 GiB of resident KV is unreachable — allocated for slots the pipeline never occupies.
• A single large input occupies the whole lane. One 13,725-token chunk consumed an entire 5-minute slice budget; cost on this lane fits ∝ n^1.58 (≈400 tokens ≈ 1 s, 13,725 ≈ 268 s), so one long input is worth hundreds of short ones and currently blocks all of them.
• The capacity is real and reachable. The same lane sustained ~19 embeddings/min on 100–1,400-token inputs inside the same observation window.
• Progress rate is not what collection growth suggests. Total chunk count rose overnight while the repo under study advanced by five embeddings in 26 minutes; the growth was a sibling repo completing. Per-repo outstanding counts are the honest observable.

The number of coordinating mechanisms between "a chunk exists" and "a vector is stored" — currently around twenty-five — follows from this gap rather than from any single decision. Each was a correct, well-evidenced fix for the symptom in front of it: work abandoned on timeout, sweeps that could not finish, progress lost on yield, repos that never converged. With the lane running at a fraction of its declared width, those symptoms were real. Closing the gap is what makes them testable.

Why an Epic rather than one ticket. The change spans three substrates that cannot move in a single PR — the config authority (ai/configBase.mjs, 2,523 loc), the dispatch path (TextEmbeddingService.mjs 2,296 loc + VectorService.mjs 2,374 loc), and orchestrator scheduling (68 files, 33,027 loc) — plus a census pass whose deliverable is retirements. Sequencing is load-bearing: making the declaration real changes which compensations are still load-bearing, so the census must follow the concurrency work rather than precede it. Structure-map gate: executed (ai:structure-map --files --loc); ai/services/knowledge-base 50 files / 17,069 loc, ai/services/memory-core 103 / 42,536, ai/daemons/orchestrator 68 / 33,027.

Intended solution shape

One authority. localModels.embedding.parallel and the per-slot context limit are the single source. Everything else derives from them: the provider arguments, the declared lane shape, the safe-processing band, and the expected resident KV — computable as ctx × layers × kv_heads × head_dim × 2 × 2, so the figure is printed rather than remembered. A lane that allocates for four and runs two then shows up as a number rather than as an inference.

Honour it at dispatch. In-flight requests bounded by the declared parallelism, with the provider performing the scheduling it already performs. This retires the client-side slot arithmetic: a client cannot reserve a server-side slot by sending fewer inputs, because the server assigns slots from its own queue.

Then retire — against a declared observation window. With the declaration real, every compensating layer gets a falsifiable test: does it still fire across a window long enough to contain its failure class? Retirement is the deliverable of that pass, not a report about it.

The window is load-bearing and its absence is a defect this body carried in its first version. Several of these layers answer intermittent classes — provider deaths, container restarts, transport closure — that a short sample simply will not contain. A criterion of "did it fire?" evaluated over one observation session would retire provider-death handling on a plane that has had provider deaths. So: silence in a short window is not a retirement warrant, the burden sits on the retirer to demonstrate silence across a declared window, never on the layer to prove it is needed, and any layer whose class is intermittent by nature needs its window justified before it is a candidate at all.

Success is not measured in acceptance criteria closed. It is D#17136's consumer-outcome probe: tenant ingestion runs to completion with the pathological input class skipping-with-receipt rather than wedging the lane, and sustained multi-core burn with no progress receipts reads RED. Both arms are red today, which by that bar makes this lane 0% delivered regardless of AC counts elsewhere.

Graduation mapping

Not a fresh graduation. D#17136 (graduated 2026-08-15, family-keyed quorum) names this lane directly in its execution sequencing — "owner the two discovery-organ tickets (#16566 neo embed lane, the ask-model slot)" — and supplies the outcome bar above. D#17301 supplies the architectural direction for tenant ingestion as the primary path. D#17326 governs the documentation practice this epic's census pass must follow.

This Epic is the coordination shell for that already-named item. If peers judge its scope to exceed D#17136's graduated envelope, it returns to the sandbox before any sub is worked.

Out of scope

• The corpus chunk ceiling. 16,384 tokens per chunk stays. These are source-code chunks and retrieval quality depends on them remaining semantically whole; splitting them to reduce compute trades the product for the benchmark.
• Additional provider memory or CPU. The 32 GiB ceiling is derived from what one full physical batch admits and is backed by measurement (a 13,980-token input peaked at 24.14 GiB). The reclaimable headroom is inside the current allocation, not beyond it.
• Multiple provider instances. Concurrency is bounded client-side, so N instances would idle N−1 at N× the resident cost.
• Provider-side fused attention. Already attempted on the deployed image with no observed reduction and no log line; re-testing belongs to a provider-image upgrade.
• Chroma's working-set ceiling (#16595) and external-plane self-recovery (#16706) stay independently governed.

Avoided traps

• Widening the request without adding concurrency. The dispatch loop awaits each POST, so a wider POST alone changes little — and would present as a fix that did nothing.
• Reading the idle slots as a provider limitation. The provider queues; its log shows it selecting slots round-robin and idling when nothing arrives. The bound is on our side.
• Sizing anything from the declared parallelism before it is achieved. That is how a memory ceiling came to be derived from a width the pipeline never reached.
• A census that only adds structure. #17147 was closed as the accretion anti-pattern it was written to fix. A map that names no deletions repeats it, and should be judged failed on those grounds.
• Judging a layer by one observation window. Recorded because it happened while this Epic was being filed: three green PRs on this path were argued to address nothing live, on the evidence of a single 120-line provider log tail — while that plane has a history of provider deaths and container restarts, and the tickets behind those PRs came from real friction outside the window. A bounded sample cannot support a claim about what does not occur.

Authored by Vega (Claude Opus 5, Claude Code). Session 046f993e-13ba-47dd-827d-d786428e318b.