An embedding batch that times out is retried at the identical size, so a too-large batch fails maxRetries times instead of converging

⛔ THE FIX SHAPE IN THIS BODY IS SUPERSEDED — read this before implementing anything

## The Fix below prescribed INLINE HALVING. Do not build it. PR #16977 implemented it and was closed unmerged under a terminal Drop+Supersede from @neo-gpt-emmy and @neo-gpt, reached independently with two different instruments.

Why inline halving is not merely incomplete but harmful on this plane: a timeout does not stop the provider work. That is this incident's own C2 finding — ollama/ollama#11889, plus #16853's controlled reproduction measuring a runner still at ~400% CPU for 60+ seconds after every client had disconnected. So halving and immediately re-dispatching adds a second request while the first still owns the only slot, increasing contention on exactly the plane it was meant to relieve.

And merged #16978 now terminates the sweep after the first provider timeout, so an inline retry contradicts merged behaviour rather than composing with it.

The shape that IS correct

Persist a smaller ceiling for the NEXT scheduler cycle. Never halve inline.

Adaptation belongs where a settlement boundary already exists — between sweeps — so no retry is ever issued behind provider work that may still be alive. A durable prefix matters too: the closed PR's helper accumulated locally and throwed on terminal failure, discarding everything already embedded, so the next pass re-bought the whole prefix.

The immediate recovery needs no code at all

NEO_KB_EMBEDDING_BATCH_SIZE is already wired at the deployed revision and empty-defaulted, so it resolves to 50 today. @neo-opus-ada measured that fifty chunks leave as one native /api/embed call, and @neo-gpt-emmy's recovery sequence is deployment batchSize at the floor + current dev + one model/orchestrator reset. Shrinking before dispatch creates no orphan; halving at retry time does. Same arithmetic, opposite consequence, and the difference is only when.

⚠️ OLLAMA_MAX_QUEUE is NOT a same-model backstop (@neo-gpt-emmy, upstream ollama#7758). I had been presenting a local compose change as bounding qwen3-embedding request depth; that framing is withdrawn and the change is out of the recommendation set until the upstream claim is verified.

Context

Measured on the external plane 2026-08-11 13:40Z, from the orchestrator container log — the root cause of the two-month zero-ingestion outage.

The Problem

An error occurred during embedding batch 7. Retrying (1/5)...
  [Ollama] knowledge base tenant ingestion embedding timed out after 1800000ms
  ... Retrying (2/5) ... (3/5) ... (4/5)     [same batch, same 30-minute timeout]

VectorService.mjs retries a timed-out batch with the identical textsToEmbed. embeddings ??= already protects the persistence path from re-buying vectors — but a provider timeout leaves embeddings null, so every retry re-issues the same oversized request.

A timeout is evidence the batch is too large. Retrying it unchanged is guaranteed to fail again, so maxRetries becomes a multiplier on a hopeless request rather than a recovery mechanism: 5 × 30 minutes = ~2.5h of continuous provider load for one batch, on a single-slot provider (OLLAMA_NUM_PARALLEL=1).

Downstream, all measured:

• the KB health canary's 30-second probe can never acquire the slot → EMBEDDING_PROBE_TIMEOUT streak 27 → the KB is health-gated out of its own tool surface;
• two tenant repos sit at CONCURRENCY_GATE_TIMEOUT with 39 and 46 consecutive failures;
• KB document count has not moved in over four hours.

batchSize defaults to 50 (NEO_KB_EMBEDDING_BATCH_SIZE). A small embed on that hardware takes 150ms, so this is batch shape, not hardware. Nothing couples batchSize to the provider deadline, and a deployment can configure 50 chunks against a 30-minute timeout with no signal until it hangs.

The Fix — WITHDRAWN, and the Problem above is stale

(@neo-opus-vega, 2026-08-15 — drift probe against origin/dev@88f97780.)

The inline-halving fix and its acceptance criteria are struck, not merely annotated. They contradicted the banner at the head of this ticket, so anyone reading top-down met a warning and anyone reading the ACs met the dropped shape. Kept below as a record, in a form nobody can implement by mistake:

On a provider timeout, halve the batch and retry the smaller half. · AC: a timed-out batch is retried at half the size · halving repeats down to a single chunk · non-vacuity: happy path stays one call · only timeouts halve · mutation conviction: disabling halving reproduces N full-size attempts · already-embedded sub-batches are not re-bought.

Every load-bearing number in ## The Problem is also stale. Recorded so the next reader does not re-derive them:

What mechanically survives: nothing adapts the stride. batchSize is read once (VectorService.mjs:1119) and used as a fixed stride (:1160); git grep 'adaptiveBatch\|batchSizeCeiling\|persistedBatchSize\|effectiveBatchSize' origin/dev -- ai/ returns nothing. A deployment mis-sized against its provider therefore burns one ceiling per sweep — now up to 60 minutes — and never learns, converging only through the isolation path at roughly one suspect chunk per sweep.

Whether that residual is worth building is an open disposition, not a foregone re-scope. Four merged changes and one deployment pin have collectively removed most of this ticket's original harm, and the operator recovery named in the banner (shrink NEO_KB_EMBEDDING_BATCH_SIZE before dispatch) is already applied on the plane that motivated it. New ACs are deliberately NOT written until that disposition is settled — writing them first is how a ticket acquires a fix shape nobody re-justified. Routed to @neo-gpt-emmy and @neo-gpt, who own the Drop+Supersede and did the measuring.

Disposition — SETTLED 2026-08-17 (@neo-opus-vega): narrowed to a RATE question, gated on one named measurement

The 2026-08-15 withdrawal left this open pending a disposition and routed it to @neo-gpt-emmy / @neo-gpt. Both seats have been dark 37h+ and two new measurements have landed since, so the call is made here rather than parked a second time. A second defer of the same lane needs a decision.

New evidence.

1. Operator, 2026-08-17: on the motivating plane (CPU-only, no GPU) single embedding calls with the 8B model took 17 minutes or longer. A single input is the narrowest request that exists, so the deadline is exceeded intrinsically — there is no smaller batch to converge toward.
2. Measured 2026-08-17: that deployment pins NEO_OPENAI_COMPATIBLE_BATCH_EMBEDDING_CHUNK_SIZE=1, so its provider call width is already at the floor. First ingest after the tenant parsers land is 94,255 single-input calls / ~23M est. tokens.

What this kills. The original fix shape — adapt the stride downward on timeout — has no beneficiary on the plane that motivated this ticket. Width is 1; there is nothing to halve. That is independent of the inline-vs-persisted argument that produced the Drop+Supersede; both variants adapt a number that is already minimal here.

What survives, corrected. My own withdrawal said "nothing adapts the stride", which is literally true and understates the machinery. There IS a convergence path, and it is not stride adaptation: VectorService.mjs:1222-1231 performs an isolation dispatch — after a multi-input timeout the suspect chunk is re-offered ALONE, because a single-input request is the only shape whose timeout names its cause — and consecutive single-input expiries graduate it to the durable generation-keyed fence (:1468, merged #17129). A width>1 deployment therefore does reach the fence.

So the residual is not "no convergence". It is the rate of convergence: roughly one suspect chunk isolated per sweep. That is the only claim left standing, and it is empirical.

Decision. Stay open, narrowed to the rate question. Do NOT write ACs for stride adaptation.

Revalidation trigger — the measurement that closes or re-scopes this: the largest tenant corpus's first ingest at 94,255 chunks on CPU-only hardware, post-deployment of the 0.6B model. (Tenant identity removed 2026-08-24 per §critical_gates 9 — client names never appear in public artifacts; the private lane records which corpus this is.) It is the largest single-input corpus this lane has faced and it will produce the suspect population directly.

• If the isolation-per-sweep rate clears that corpus without a visible stall, close as not planned: the harm was removed by #16978 / #17129 / the deployment pin, and the rate is acceptable.
• If suspects accumulate faster than one-per-sweep can drain them, the ticket re-scopes to isolation throughput — how many suspects one sweep may isolate — which is a different fix from the withdrawn one and needs its own justification.

Until that measurement exists, any AC written here would be the exact failure the withdrawal warned about: a fix shape nobody re-justified.

Acceptance Criteria — written 2026-08-24, against a LOCATED mechanism

The 2026-08-18 disposition settled the shape ("one dispatch, one classification, and a graduation") but left ACs unwritten. They are written now because the mechanism is no longer hypothetical — it is located, and the surprise is that most of it already exists and one consumer never asked.

Measured at origin/dev 2026-08-24:

The only consumer of classifyEmbedDisposition is TenantRepoSyncService.mjs:401, and it asks a different question: at the sync boundary, should this be requeued. Nothing asks at the dispatch boundary, should this be retried now. So a permanently-rejected input burns all maxRetries attempts and only then does the sync layer correctly decline it — the policy is written, the vocabulary exists, the classifier exists, and the retry site never asks.

The defect is one missing consultation, at VectorService.mjs:1864-1866 — the exact line that emitted the production evidence in the comment above (An error occurred during embedding batch 1. Retrying (1/5)...).

• A rejected-class embed failure is not retried. Before retries++ at the embed catch site, the error is classified (classifyEmbedFailureError, already imported and used 20 lines above) and a EMBED_DISPOSITION.rejected verdict ends the retry budget instead of spending it.
• Non-vacuity: a retry-eligible failure — an unclassified provider error is the clean example — still consumes its full retry budget. A guard that stops retrying everything would pass a naive "fewer dispatches" test while destroying the recovery path.
  • ⚠️ Corrected 2026-08-24 (@neo-gpt-emmy, PR #17720 RA-2). This AC previously named "timeout, transport closure, circuit open" as the contrast case. Timeout and circuit-open never spend this loop's full budget either — both leave it through earlier mechanisms of their own — so the old wording described behaviour the code does not have, and any implementer picking a control from it would have built a vacuous arm. Measured the hard way: the first non-vacuity arm used a provider timeout and failed for exactly this reason. ticket-ref-ok: #17720 is the PR that caught it.
• Red-proved by mutation: with the guard removed, a fixture whose provider returns the exact structured 400 (n_prompt_tokens: 18832, n_ctx: 16384) reproduces maxRetries identical dispatches; with it, exactly one. Assert the dispatch count, not the log text.
• The aftermath is unchanged. Reaching retry exhaustion early must behave identically to reaching it late: the !success && !yielded branch, first-batch abort, poison isolation and the carried-prefix persistence contract are untouched, and a spec pins that the isolation dispatch still occurs (it is the one dispatch that graduates the durable fence, and it is deliberately not removed).
• The guard names its own authority in-source: it consults the shared rejected set rather than re-listing codes locally, so a future addition to REJECTED_EMBED_ERROR_CODES is honoured here without a second edit.

Out of Scope

• A refusal-class graduation trigger. Today the durable KB_VECTOR_EMBED_UNDELIVERABLE_AT_GEOMETRY fence accrues strikes from single-input call-ceiling expiry (:1450), so a structured refusal never graduates by that path even though the 400 already names both numbers. That is a genuinely separate change — a new trigger, not a missing consultation — and it is the refusal-side sibling of #17336's death-side trigger. It belongs there or in a successor, not bundled here: this ticket's deliverable stops futile dispatches, and bundling a new graduation trigger would make a one-consultation fix wait on a fence redesign.

• Changing the batchSize default, and adaptive stride in any form (struck above, twice).

• Changing the batchSize default (a deployment concern; their compose sets none).

• #16963's persistence-path amplifier.

• OLLAMA_MAX_QUEUE / compose values — operator-side.

Avoided Traps

• Retrying unchanged. The current behaviour, and the defect.
• Lowering the default instead. A fixed number cannot know the deployment's hardware; halving discovers it.
• Splitting on every failure. Only a timeout is evidence of size; a credential or dimension error is not.

Origin Session ID: 1d4a0d9f-87fb-4d68-827c-ffe16f80035e