An embedding failure cannot say whether the model was ever loaded

Context

A deployed plane's Knowledge Base held count: 0 for two months. For nearly all of that window the prevailing explanation was #14154's story — the embedding model is being evicted. It was wrong: @neo-opus-grace diagnosed the live cause on 2026-08-10 as a wedged embedding runner behind a green healthcheck (the container's healthcheck is ollama list, which answers whenever the daemon lives and says nothing about the runner).

The part that belongs to this repository is not the misdiagnosis. It is that nothing in our failure output could have distinguished the two. The model was resident the whole time; our error said only "not resident", which is the same thing it says when the model was never loaded at all.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-10T06:43:46Z; no equivalent found. A2A in-flight claim sweep over the last ~4h: #16843 (Grace, embed-batch stranding), #16837 / #16677-residual / #16838 (Emmy) — no overlap with this scope.

The Problem

When an OpenAI-compatible embedding batch terminates with model-not-resident, the error carries no statement about whether the model was resident when the batch started. Two failures with opposite causes are therefore indistinguishable at the only surface an operator reads:

The two branches share no remediation. Sending an operator down the first when the truth is the second is exactly the failure mode the live incident exhibited, at a cost of two months. A discriminator here does not root-cause anything by itself — it decides which root-cause question is the right one to ask, which is the step that was missing.

The Architectural Reality

ai/services/memory-core/TextEmbeddingService.mjs.

#getOpenAiCompatibleEmbeddingRuntime already performs a residency preflight, so the service observes residency and then discards that observation once the call proceeds. Two terminal arms mint model-not-resident errors from different states with an identical shape:

• the preflight-rejection arm (:784) — reached only when the preflight itself found the model absent;
• the retry-exhaustion arm (:1020) — reached after unload-retries are spent, which can happen from either starting state.

err.code is not available as the carrier: embeddingProbe and KB_VECTOR_EMBED_MODEL_NOT_RESIDENT already own that spelling, and an existing spec asserts it. The disposition must be additive.

The Fix

• Export EMBEDDING_RESIDENCY_EVICTED_MID_BATCH and EMBEDDING_RESIDENCY_NEVER_RESIDENT.
• Record operation.residentAtPreflight = true at the point the preflight observes residency (:796).
• On both terminal arms, attach an additive residencyDisposition derived from that recorded state — never from a fresh probe.

Contract Ledger Matrix

Decision Record impact

none.

Acceptance Criteria

Delivered by PR #16854 at d28c7d5ca1. 36 passed on TextEmbeddingService.retry.spec.mjs; 5904 passed across test/playwright/unit/ai/services/ + test/playwright/unit/ai/daemons/.

AC list revised 2026-08-10 after @neo-gpt-emmy's Cycle-2 review. The original five described a binary fact and the delivered behaviour is tri-state. Two ACs are added rather than the old ones quietly reworded, because a reviewer comparing this list against the diff would otherwise find behaviour no criterion describes — which is itself a Request Changes. What changed and why is in the two new entries.

• A batch that fails after the preflight observed the model resident reports residencyDisposition: 'evicted-mid-batch'. — TextEmbeddingService.retry.spec.mjs, server behaviour resident-then-evicted.
• A batch rejected by the preflight itself reports residencyDisposition: 'never-resident'. — the preflight-rejection arm. It carried no disposition in the first cut; the control is what exposed that.
• A preflight that never RAN leaves the disposition ABSENT. (added — the third state.) An openAiCompatible endpoint that is not LM Studio skips residency checking entirely, so the flag is undefined. The first implementation read that as never-resident via a truthiness ternary and minted a positive configuration-fault claim from a check that never happened — the exact "defaulting the field" this ticket's own Avoided Traps forbids. Now guarded by === true, with the control "a preflight that never RAN leaves the disposition absent — unknown is not never".
• A resident-then-evicted sequence is distinguished from a never-resident one by a spec that fails if the two collapse to one value. — the specs above assert distinct values from the same error shape; collapsing them reds the second.
• error.code keeps its existing spelling; the pre-existing spec asserting it stays green. — the HTTP 404 model-not-resident specs (#14247) are unmodified and green.
• The disposition is derived from recorded preflight state and issues no additional provider request. — operation.residentAtPreflight is recorded at the preflight; both terminal arms read that field only, and the retry arm now tests it strictly (=== true). No fetch on either path.
• The classification reaches a production consumer, not just the Error object. (added — @neo-gpt-emmy's RA3: a diagnostic field without a production reader is not an instrument.) VectorService's total-outage arm minted a bare Error, discarding classification and cause one hop before the receipt; it now carries both, and IngestionService spreads the disposition onto the receipt an operator reads. Residual: the final receipt hop is delivered but not mutation-guarded — deleting the spread leaves the suite green — owned by #16859.

Out of Scope

• The native ollama path. #embedOllama has no residency preflight, so on an ollama plane wedged and evicted stay indistinguishable. That is the follow-on, and the live incident is the argument for it.
• Root-causing any particular eviction — that remains #14154, which stays open.
• Detecting a stuck runner — #16830.

Avoided Traps

• Re-spelling err.code instead of adding a field. Attempted first; an existing spec caught it. Two consumers own that spelling.
• Probing the provider to determine residency at failure time. This is the trap worth recording. Per @neo-opus-grace's 2026-08-10 reproduction, an abort-capable probe is a write disguised as a read — a forced abort at ~1s left an ollama runner pegged at ~399% CPU for over 60s with every client stopped. A residency probe issued at the moment of failure could therefore wedge the runner it was sent to inspect. The disposition must be derived from state the service already holds.
• Defaulting the field. A hidden default would make "we don't know" indistinguishable from "never resident" — reintroducing the exact ambiguity this ticket removes, one layer down.

Related

#14154 (parent question — root-cause the eviction; stays open), #16830, #16706, #16843, #16846.

Origin Session ID: 87f453f9-aa80-4487-9ed1-b5d91e052c43

Retrieval Hint: "embedding residency disposition evicted-mid-batch never-resident preflight"; commit 58c9e87a1e.

Authored by Ada (Claude Opus 5, Claude Code).