A chunk that kills the embedding provider never graduates to undeliverable

Context

Observed live on a deployment running the openAiCompatible embedding lane (llama.cpp server, qwen3-embedding-0.6b), 2026-08-18. Its container memory ceiling was set below what one admitted chunk requires, so the kernel OOM-killed the engine on the first band-sized batch. restartCount reached 36 in ~50 minutes with 14 unplanned restarts inside one 15-minute window, and the knowledge base stayed at 152 vectors against a corpus of roughly 93,500 semantic units.

The ceiling itself was a deployment error and is fixed there. This ticket is about what neo did in response.

⚠️ Re-censused 2026-08-19 (first) — the live trigger is DISCHARGED. #17343 / #17347 merged and now key admission on the engine slot in the unit the engine counts, so the oversized chunk that killed the provider is refused at admission rather than dispatched. It becomes an admission rejection, not a provider death. Verified present in the deployed pin by content (the merges were squashes, so PR-head ancestry reports false).

What survives is the general case: a chunk within the admitted band that kills the provider anyway — a lowered ceiling, a different engine, or the squared memory curve at a band that is correct but unlucky.

And the safety precondition this ticket needed is now satisfied. #17345 merged: the poison generation derives from the resolved admission band, so a geometry graduation is reversible by the repair — fix the ceiling, the band moves, the generation changes, every geometry-fenced chunk is re-offered. That converts the "silently shrink the corpus" trap below from durable content loss into deferral-until-repaired, which is what the word geometry is supposed to mean. Before #17345 this graduation was a one-way door and should not have shipped.

⚠️ Re-censused 2026-08-19 (second) — I MEASURED THIS TICKET'S OWN MECHANISM AND IT INVERTED. The title survives; the stated harm does not.

Both this Context and the first re-census asserted that a provider-killing chunk is "re-offered forever" and that the repository "walks its consecutiveFailures up until backoff suppresses it permanently." That is false, and I am replacing it rather than softening it. The chunk stops being offered at the second sweep — it is fenced by the pre-existing isolateFirstFailedBatch, as content poison, carrying a transport-death reason code.

So the timeout path and the death path both converge. The death path converges on the wrong verdict class, which is the precise harm #17129 built the geometry/content distinction to prevent. The ticket's title — a chunk that kills the embedding provider never graduates to undeliverable — remains literally true: it never reaches KB_VECTOR_EMBED_UNDELIVERABLE_AT_GEOMETRY. It reaches a false content verdict instead.

A false quote of our own source is also removed here. The prior body attributed to VectorService.mjs:1011 the sentence "a dead provider proves nothing about content and must never be bisected into a poison disposition." No such sentence exists. Line 1093 says "A Chroma write error proves nothing about content and must never be bisected into a poison disposition" — a different subject entirely — and the dead-provider sentence at 1006 says something narrower, quoted correctly below. Prior line citations :920-923 and :1011 were also off; corrected to HEAD line numbers throughout.

Separating observation from inference: the OOM attribution is inferred from memory pinned at 99.94% of cap (two authoritative critical evidence facts, recoveryClass: exhaustion), a restart count that only climbs, and container logs in which every incarnation ends mid-launch_slot_ with no error, assertion or exit line. inspect.state.oomKilled reads empty because state describes the live incarnation, not the one that died, so the flag is not available as proof. The second re-census above is measured, not inferred: the fencing was observed at sweep 2 with a transport reason code on a poison entry.

⚠️ Re-censused 2026-08-19 (third) — the recovery-probe mechanism is SUPERSEDED, measured, not reasoned. The probe was unreachable once the suspect is the last chunk (strikes frozen at 1 across six sweeps, chunk fenced by nothing) and cost a provider request per death on a saturated lane. Replaced by the accepted-then-died classification already latent in KB_VECTOR_EMBED_TRANSPORT_CLOSED, which supplies the same liveness half for free. The Fix and the ACs below are updated; the graduation is now reachable from the failure path as well as the success path.

The Problem

Neo already has the right disposition for this shape. #17129 built it: a chunk the current geometry cannot deliver graduates to KB_VECTOR_EMBED_UNDELIVERABLE_AT_GEOMETRY, stops being offered, and the corpus advances past it. VectorService.mjs is explicit that this is content-neutral — "healthy content the current geometry cannot deliver" — and deliberately distinct from a content-poison disposition.

That graduation is gated on timeout-class evidence only. The strike counter increments on single-input call-ceiling expiry, and its own log line names the evidence: "after N consecutive single-input call-ceiling expiries (~X tokens against a Yms ceiling)". #17129's title says the same thing — ceiling-fired-twice must classify as undeliverable.

An OOM-killed provider does not time out. It refuses the connection, or drops the socket mid-request. We observed KB_VECTOR_EMBED_CONNECTION_REFUSED live, and KB_VECTOR_EMBED_TRANSPORT_CLOSED on the re-measurement. Neither accrues a strike, so the suspect never graduates to geometry.

What happens instead — measured, and the correction that reshapes this ticket. isolateFirstFailedBatch fences the suspect as content poison on its very next isolation pass. Two facts make that a misclassification rather than a judgement call:

1. isPoisonIsolationForbidden (VectorService.mjs:911) carries no death term. Its terms are signal.aborted, isEmbeddingBatchYieldError, AbortError, ABORT_ERR, KB_VECTOR_EMBED_PROVIDER_CIRCUIT_OPEN, and isProviderTimeoutCode. Its own JSDoc enumerates the content-neutral categories as timeout, abort/circuit, and cooperative yield — a list that reads as complete and omits provider death. So a death is permitted to produce a poison disposition.
2. The stored record contradicts itself. createPoisonEntry sets reasonCode: classifyEmbedFailureError(error) (:1044-1048), so the resulting poison entry carries a transport-death reason code. A poison verdict whose own reason code names a socket failure is a defect visible in the data, without needing to re-derive intent.

The paired-control protection exists, and it is precisely what makes a killer chunk look poisonous. The bisection path takes real care here: at :1070 it issues a fresh control dispatch immediately before attributing, with the reasoning at :1067 — "Paired evidence at the decision boundary. The earlier control success is not enough: a provider can die during the split walk, and quarantining everything after that point would turn a transient outage into durable content loss." If that fresh control throws, the throw propagates and nothing is quarantined.

That protects the remainder, which is exactly what the contract at :1006 claims — "a provider that dies during isolation therefore aborts instead of quarantining the remainder." It does not protect the trigger. For a chunk that kills the provider, the sequence is: fresh control succeeds (provider demonstrably alive) → suspect dispatched → provider dies → non-forbidden error → poison. Liveness-immediately-before-this-input is the evidence pattern a killer chunk always produces, so the strongest available exculpatory check is the one that convicts it.

A second asymmetry was claimed here and is withdrawn — the design is correct. An earlier revision of this section argued that the failedInputs.length === 1 path (:1099-1109) wrongly lacks the fresh control the bisection path takes at :1070. Reading :1036-1038 refutes it: isolateFirstFailedBatch issues its control up front, awaited and uncaught, so a dead provider throws before anything is attributed. On the singleton path that control is one request old at the moment of attribution — already fresh. The bisection path refreshes because its recursive walk moves the decision away from the original control, which is exactly what its :1067 comment says. The two paths are not in comparable positions and the asymmetry is justified. No change is needed there.

The Architectural Reality

All line numbers are at HEAD, verified by reading the committed file rather than a working tree.

• ai/services/knowledge-base/VectorService.mjs:911 — isPoisonIsolationForbidden, the guard, and its complete term list. No death term.
• ai/services/knowledge-base/VectorService.mjs:1001 / :1006 — isolateFirstFailedBatch's summary and the dead-provider contract, whose subject is the remainder.
• ai/services/knowledge-base/VectorService.mjs:1044-1048 — createPoisonEntry, which stamps the failure's own classified code onto the poison entry.
• ai/services/knowledge-base/VectorService.mjs:1067-1072 — the fresh paired control on the bisection path, and the reasoning for why an earlier control success is not enough.
• ai/services/knowledge-base/VectorService.mjs:1082 and :1108 — the two poison-attribution sites, each gated only by the guard.
• ai/services/knowledge-base/VectorService.mjs:1093 — the Chroma-write-error sentence, quoted correctly here because the prior body mis-attributed it.
• ai/services/knowledge-base/VectorService.mjs:1099 — the singleton branch that skips the paired control.
• ai/services/knowledge-base/IngestionService.mjs:512 — KB_EMBED_BATCH_SKIPPED, raised once retries are exhausted.
• The tenant-repo-sync scheduler consumes the failure as a repo-level consecutiveFailures increment; per-repo backoff is 2^consecutiveFailures × cadence capped at 30 minutes, from persisted state.

The distinction this ticket needs is provider death that reproduces against one isolated input under a freshly recovered provider. That is not ambient provider instability, and it is not a content claim — it is evidence about deliverability at the current geometry, which is exactly what UNDELIVERABLE_AT_GEOMETRY already means.

The Fix

Sequencing constraint, and it is why this stays one ticket. Adding a death term to the guard, alone, is a regression: today the killer chunk at least stops being offered (with a false verdict), so the corpus advances. Forbid death-induced poison without providing the death-class graduation and the chunk is fenced by nothing — restoring the re-offered-forever behaviour this ticket originally, and wrongly, claimed already existed. The guard fix and the graduation must land together.

1. Add provider-death codes to isPoisonIsolationForbidden (:911), so a death during isolation throws rather than attributing. This makes the :1006 contract cover the trigger and not only the remainder, and it stops poison entries from being stamped with transport reason codes.

2. Classify provider death (connection refused, socket reset, unexpected EOF mid-request) as its own failure class, distinct from both timeout and provider-refusal.

3. Accrue a death-class strike only when a single-input dispatch kills the provider and the provider subsequently returns healthy — the recovery is what makes the correlation attributable rather than ambient. A death with no recovery accrues nothing.

   The liveness signal comes from the FAILURE CODE, and it is free — superseding this ticket's earlier prescription. ECONNRESET / EPIPE / UND_ERR_SOCKET all classify as KB_VECTOR_EMBED_TRANSPORT_CLOSED, and every one of them requires an established connection: a peer cannot reset, or close under our write, a connection it never accepted. So the code carries its own proof that the provider was answering when the request left, and that this input was the one in flight. That is the whole "provider was alive AND this killed it" pair, at zero extra provider requests. KB_VECTOR_EMBED_CONNECTION_REFUSED is the opposite — nothing was listening, so it proves the provider was already dead and attributes nothing; it stays an unattributable pending observation.

   What this replaces, and why. The first prescription was a recovery probe: record a pending death, convert it once a later dispatch succeeds. It is unreachable in the case this ticket exists for. Once the suspect is the only chunk left in the corpus, nothing is dispatched after it, no success is ever observed, and the counter freezes one strike below threshold forever. Measured on the fixture: strikes stuck at 1 across six sweeps with the chunk fenced by nothing. The probe variant also cost one provider request per death, which is not free on a CPU-saturated embedding lane. Both problems dissolve when the evidence comes from the failure itself. It remains deliberately not embeddingRecoveryProbe — #17337 measures why that probe reports healthy two seconds after the lane's own lastErrorAt.

   Stated limit, unchanged: accepting a request proves liveness, not capacity at the suspect's size. That is sufficient for a disposition that says geometry rather than content, and only because #17345 makes it reversible. A single reset is also only a sample — a network blip can reset a connection with the input blameless — so the strike threshold, not the predicate, is what gates a graduation.

4. Graduate at threshold to KB_VECTOR_EMBED_UNDELIVERABLE_AT_GEOMETRY, reachable from both provider outcomes. The success path converts a pending observation; the failure path graduates directly when the failure itself proved liveness. A success-gated graduation alone is unreachable exactly when it is needed, for the same reason the probe was.

5. Surface death-class strike progress in the ingestion summary, so convergence is visible instead of consecutiveFailures climbing with no explanation.

Contract Ledger Matrix

Decision Record impact

none — this extends a disposition #17129 established rather than challenging any ADR.

Acceptance Criteria

• Red-proof on the misclassification: against dev, a fixture whose input kills a stubbed provider must produce a poison entry whose reasonCode is a death code. That assertion must FAIL after the change. A timeout fixture proves nothing here — timeouts are already forbidden.
• A provider death during isolation no longer produces a poison entry, on both the bisection path and the failedInputs.length === 1 path. Asserted per path, because they attribute at two different sites (:1082 and :1108).
• A single-input dispatch whose failure classifies as KB_VECTOR_EMBED_TRANSPORT_CLOSED accrues a death-class strike against that input, with no additional provider request spent to prove liveness.
• A KB_VECTOR_EMBED_CONNECTION_REFUSED death accrues no strike and quarantines nothing — nothing was listening, so it attributes nothing to the input, and the corpus keeps the chunk offerable rather than fencing it under either verdict.
• Graduation is reachable when the suspect is the last unembedded chunk. Asserted on a fixture where the killer is the only work left, so no dispatch after it can succeed. A suite that only covers a corpus with surviving work would pass while the automaton freezes.
• At threshold, the suspect graduates to KB_VECTOR_EMBED_UNDELIVERABLE_AT_GEOMETRY with a receipt recording both the suspect's and the recovering input's size, and is no longer offered until the embedding generation changes.
• Sequencing guard: a test asserts that a provider-killing chunk is fenced by some disposition at the end of a sweep. This must pass before and after, and it is what catches the regression where the guard fix ships without the graduation.
• A corpus containing one provider-killing chunk completes ingestion of every other chunk in the same sweep — asserted on kind and count, not merely on absence of error.
• Death-class strike progress appears in the ingestion summary.
• The :1006 contract's existing behaviour for the remainder is unchanged, with a regression guard: a provider that dies mid-bisection still aborts rather than quarantining what follows.

Out of Scope

• Changing batch sizing or adaptive stride. That is #16972, narrowed to isolation rate and gated on first-ingest evidence.
• The deployment's memory ceiling — a deployment concern, already corrected there.
• Content-poison classification for genuine content failures. (Narrowed 2026-08-19: the prior body put "any change to content-poison classification" out of scope, which is incompatible with the measured mechanism — the fix necessarily changes what may be classified as poison. What stays out of scope is the disposition for failures that are genuinely content-caused; only death-caused attribution changes.)
• The recovery actuator's action set — separate ticket.

Avoided Traps

• Asserting the harm without measuring it. This ticket twice claimed the killer chunk is "re-offered forever." It is fenced at sweep 2, as poison. The corrected harm — a false content verdict on healthy content — is both narrower and worse, and it is what the fix has to target. Recorded because "it never converges" is what the symptom looks like from outside, and it is what a successor would re-derive.
• Quoting our own source from memory. The prior body attributed the Chroma-write sentence at :1093 to provider death, and cited :920 / :1011 for text at :911 / :1006. A misquote of the contract you are reasoning about is worse than no quote, because it manufactures agreement.
• Claiming a second defect without reading the control acquisition. An earlier revision of this amendment asserted the singleton path wrongly lacks a fresh paired control, and carried an AC and a Contract Ledger row for fixing it. :1036-1038 refutes it — the up-front control is one request old at that attribution, so it is already fresh, and the bisection path refreshes only because its walk moved the decision away. Withdrawn before any code was written, and recorded because "one path has a check the other lacks" reads as a defect until you check what sits above both.
• Reading the paired control as protection here. It is real protection for the remainder and it is well-reasoned at :1067. For the trigger it does the opposite: liveness-immediately-before-the-suspect is exactly what a provider-killing input produces.
• Fixing the guard alone. That removes the wrong fence and restores no fence. Hence the sequencing constraint and its AC.
• Do not widen the timeout strike counter to "any failure". That would let ambient provider instability graduate healthy chunks to undeliverable and silently shrink the corpus. The recovery-confirms-correlation requirement is what keeps the evidence attributable.
• Do not treat the container's restart count as the trigger. Neo cannot see it in the general case, and a deployment may not restart at all — the evidence has to come from the request path.

Related

• #17129 — established the UNDELIVERABLE_AT_GEOMETRY graduation for the timeout path; this fills the death-class gap beside it and repairs the guard that lets death reach a content verdict.
• #16972 — identical-size retry on timeout; adjacent but distinct trigger.
• #16843 — one failing batch stranding every chunk after it; the ancestor of the skip-rather-than-abort behaviour.
• #17048 — batch width bounded to leave an engine slot admissible.
• #17044 — self-heal futility breaker; the actuator-side sibling.
• #17343 / #17347 — discharged the live trigger by refusing the oversized chunk at admission.
• #17345 — made a geometry graduation reversible; the safety precondition for this ticket.
• #17337 — why embeddingRecoveryProbe is not the recovery signal.

Live latest-open sweep: checked latest 20 open issues at 2026-08-18T09:29:25Z plus a six-term state=all title sweep; #17129 is closed and timeout-scoped, #16972 is open and retry-size-scoped, neither covers death-class graduation.

Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2

Amended 2026-08-19 in session 8cbd588b-be06-4a56-9997-1058f2a3a07b — mechanism re-measured, false quote removed, line citations corrected to HEAD, scope narrowed, ACs restated.

Retrieval Hint: query_raw_memories("embedding provider death bisected into content poison isPoisonIsolationForbidden no death term paired control convicts the trigger undeliverable geometry")