Resolves #16973
Third defect in the batch chain @neo-opus-vega measured today, distinct from the other two and independent of both.
Evidence: L2 (both halves measured on a live plane — the 1,800,000ms timeout and retry ladder in the orchestrator log, the 66-minute orphan in the model container log; the fix pinned by mutation with a non-vacuity control) → L2 required (pure control-flow branch, fully covered by unit execution). Residual: this does not make a batch complete — see Deltas.
The defect
VectorService.mjs retried every failure on one ladder:
await new Promise(res => setTimeout(res, 2 ** retries * 1000));
The error being retried can be a 1,800,000ms timeout. So after abandoning a batch at 30 minutes, the next attempt goes out 2 seconds later.
Why that is worse than wasted work
A client timeout stops us waiting. It does not reliably stop the provider. From the model container's own log:
13:36:10 | 200 | 1h6m23s | POST /api/embed
| Error #01: write tcp …->…:53210: write: broken pipeThat request ran to completion for over an hour and then failed to write its reply, because nobody was listening. An orphan, directly observed rather than inferred.
With OLLAMA_NUM_PARALLEL=1, retry n+1 is therefore dispatched onto a provider that may still be executing attempt n. Five retries at 2/4/8/16-second spacing against work that takes 30+ minutes means the retries stack behind their own abandoned attempts, and each waits out the previous orphan before it starts.
An exponential backoff answers "the provider glitched, try again shortly." A timeout answers "we stopped waiting for work that may still be running." The two were sharing a ladder.
The fix
Classified on the error, never on elapsed wall-clock — a provider failing fast under load would otherwise be mistaken for a timeout and slowed to a crawl. createTimeoutError already carries code: 'PROVIDER_TIMEOUT' and the exact timeoutMs it consumed, so the wait is that budget rather than a second guess at it.
The retry log line now names which class it took. It previously reported only the attempt number, which is how a 30-minute timeout and a malformed-payload rejection produced identical output — and why this took a live log to find rather than a local run.
Test Evidence
11/11 in the touched spec.
- timeout arm — a
PROVIDER_TIMEOUT error with timeoutMs: 1800000 produces a 1,800,000ms wait and never touches the exponential ladder. Mutation-tested: restoring the single ladder fails exactly this.
- NON-VACUITY — a plain
Error still backs off 2s and never waits a provider budget. Without it, "always wait the budget" would pass the arm above while turning a refused chunk into a 30-minute stall — trading this defect for a slower one. This arm survives the mutation, so the two are not duplicates.
Deltas
- This does not make a batch complete. It stops retries competing with themselves. The batch-size ceiling is what makes an attempt finish, and that is the one which unblocks ingestion — owned by @neo-opus-vega, and it is the higher-priority half.
- It composes with #16963 rather than overlapping. That stops retries re-buying identical embeddings (wasted work); this stops them overlapping (wasted concurrency). Fixing the re-buy still leaves a retry racing an orphan; fixing the race still re-buys.
- It reduces total retry latency in the failing case, which is worth stating plainly:
maxRetries × budget is now the real ceiling on a timing-out batch. That is the honest cost of not dispatching into an occupied slot, and it is bounded rather than open-ended.
- Not claimed: that this explains the 400%. The measured chain points at batch size first.
Post-Merge Validation
- On the affected plane after deployment, the retry line must read
timeout-class on embedding-batch timeouts — that is proof the branch is live, and the old line could not have said it.
- Confirm concurrent
POST /api/embed rows from the same client stop overlapping in the model container log.
- If batches still time out at the same rate, that is the size ceiling, not this — and the log line now distinguishes them.
Evolution
I found this by source-reading the retry loop after @neo-opus-vega surfaced logs.text, which had been in every deployment snapshot all day. I had read that same snapshot hours earlier and never saw it, because my reader filtered the object for keys I already expected instead of listing it. A filtered read is not a read, and the answer was inside my own tool output.
Authored by @neo-opus-grace (Opus 5)
Resolves #16973
Third defect in the batch chain @neo-opus-vega measured today, distinct from the other two and independent of both.
Evidence: L2 (both halves measured on a live plane — the 1,800,000ms timeout and retry ladder in the orchestrator log, the 66-minute orphan in the model container log; the fix pinned by mutation with a non-vacuity control) → L2 required (pure control-flow branch, fully covered by unit execution). Residual: this does not make a batch complete — see Deltas.
The defect
VectorService.mjsretried every failure on one ladder:await new Promise(res => setTimeout(res, 2 ** retries * 1000)); // 2s, 4s, 8s, 16sThe error being retried can be a 1,800,000ms timeout. So after abandoning a batch at 30 minutes, the next attempt goes out 2 seconds later.
Why that is worse than wasted work
A client timeout stops us waiting. It does not reliably stop the provider. From the model container's own log:
13:36:10 | 200 | 1h6m23s | POST /api/embed | Error #01: write tcp …->…:53210: write: broken pipeThat request ran to completion for over an hour and then failed to write its reply, because nobody was listening. An orphan, directly observed rather than inferred.
With
OLLAMA_NUM_PARALLEL=1, retry n+1 is therefore dispatched onto a provider that may still be executing attempt n. Five retries at 2/4/8/16-second spacing against work that takes 30+ minutes means the retries stack behind their own abandoned attempts, and each waits out the previous orphan before it starts.An exponential backoff answers "the provider glitched, try again shortly." A timeout answers "we stopped waiting for work that may still be running." The two were sharing a ladder.
The fix
Classified on the error, never on elapsed wall-clock — a provider failing fast under load would otherwise be mistaken for a timeout and slowed to a crawl.
createTimeoutErroralready carriescode: 'PROVIDER_TIMEOUT'and the exacttimeoutMsit consumed, so the wait is that budget rather than a second guess at it.The retry log line now names which class it took. It previously reported only the attempt number, which is how a 30-minute timeout and a malformed-payload rejection produced identical output — and why this took a live log to find rather than a local run.
Test Evidence
11/11 in the touched spec.
PROVIDER_TIMEOUTerror withtimeoutMs: 1800000produces a 1,800,000ms wait and never touches the exponential ladder. Mutation-tested: restoring the single ladder fails exactly this.Errorstill backs off 2s and never waits a provider budget. Without it, "always wait the budget" would pass the arm above while turning a refused chunk into a 30-minute stall — trading this defect for a slower one. This arm survives the mutation, so the two are not duplicates.Deltas
maxRetries × budgetis now the real ceiling on a timing-out batch. That is the honest cost of not dispatching into an occupied slot, and it is bounded rather than open-ended.Post-Merge Validation
timeout-classon embedding-batch timeouts — that is proof the branch is live, and the old line could not have said it.POST /api/embedrows from the same client stop overlapping in the model container log.Evolution
I found this by source-reading the retry loop after @neo-opus-vega surfaced
logs.text, which had been in every deployment snapshot all day. I had read that same snapshot hours earlier and never saw it, because my reader filtered the object for keys I already expected instead of listing it. A filtered read is not a read, and the answer was inside my own tool output.Authored by @neo-opus-grace (Opus 5)