An embed failure's cause is known inside the process and never minted into a durable code

Context

Successor to #16647, carved out during review of PR #16657, then re-scoped after intake by @neo-gpt (comment). The original framing bundled three different input classes under one label; that correction is folded in below and is the reason this body reads differently from its first revision.

#16647 was filed from a live external deployment whose four tenant repos all failed with:

lastErrorCode       = KB_TENANT_REPO_SYNC_SYNC_FAILED
lastSourceErrorCode = KB_VECTOR_EMBED_FAILED
lastIngestedRev     = null

PR #16657 fixed a real and separate defect: provider-classified failures (EMBEDDING_PROBE_TIMEOUT, ABORT_ERR) were truthy, so the || 'KB_VECTOR_EMBED_FAILED' fallback never fired for them — the provider's own code was recorded, then dropped by the writer-side ^KB_ filter, landing lastSourceErrorCode as null.

That fix does not change the receipt above, and this ticket carries what it left.

The Problem

The cause of an embed failure is very often known inside the process and never minted into a durable code. Three distinct shapes reach the ingest boundary, and the first revision of this ticket wrongly treated them as one:

1. Absent code, internally recognized cause

ai/services/memory-core/TextEmbeddingService.mjs:574 — the model-not-resident check:

throw new Error(`TextEmbeddingService: LM Studio embedding model '${model}' is not resident under its configured identifier; observed=${observedIds}`)

No code. The service knows the cause precisely, and puts it in a message string that also interpolates the configured model identifier and up to five observed ids — so the message is both the only carrier of the diagnosis and unsafe to surface. :781 compounds it: the retry detector recognizes model-not-resident by message substring (HTTP 404, Model was unloaded), acts on that knowledge, and still mints nothing durable.

2. Code-bearing, but outside our trusted vocabulary

:755 — the transport error pass-through:

req.on('error', err => rejectOnce(isCallerAbortError(err, signal) ? getEmbeddingAbortError(signal, operationLabel) : err))

A refused connection arrives carrying a real runtime code — ECONNREFUSED, ENOTFOUND, ECONNRESET. This is not an absent-code case. The code exists, is meaningful, and is simply not in any vocabulary we own — so classifyEmbedFailureCode correctly declines to pass it through, and the cause is lost for a different reason than shape 1.

3. Genuinely unclassifiable

Everything else. KB_VECTOR_EMBED_FAILED is the honest answer here and should keep meaning exactly this — and only this.

Why the split is load-bearing. Shapes 1 and 2 need opposite repairs: shape 1 needs a code minted at the site that already knows the cause; shape 2 needs a trusted translation of a foreign code we must never echo. Treating them as one defect produces a fix that solves neither, and an AC that a synthetic pair of bare errors can satisfy without touching the production connection-refusal path.

describeEmbeddingProbeFailure (ai/services/memory-core/HealthService.mjs:208) is not the vocabulary to lift unchanged, contrary to this ticket's first revision: every cause outside a four-item list collapses to provider-failure:EMBEDDING_PROVIDER_ERROR, so it maps both shapes above to the same string and cannot satisfy AC1 as written.

The Architectural Reality

• ai/services/memory-core/TextEmbeddingService.mjs:574 — shape 1; :781-786 — the message-substring detector that already discriminates and mints nothing; :755 — shape 2.
• ai/services/knowledge-base/VectorService.mjs:1144 — the single coded path (KB_SYNC_VOLUME_EXCEEDED); the rest of embed / embedChunks propagates uncoded.
• ai/services/knowledge-base/helpers/embedFailureClassification.mjs — the boundary shipped by PR #16657. Its closed upstream-code map translates both the source-owned generic model-residency code and Node's foreign transport code into KB-owned literals; INTERNAL_EMBED_ERROR_CODES remains limited to codes already minted by KB-owned layers.
• snapshot.tenantRepoSync.repos[].lastSourceErrorCode — the remotely-readable surface; codes only, by a deliberate credential boundary.

Contract Ledger

Acceptance Criteria

• A connection refusal (code-bearing, e.g. ECONNREFUSED) and a model-not-resident failure (bare Error) are distinguishable from snapshot.tenantRepoSync.repos[].lastSourceErrorCode alone, with no shell access.
• Each specimen matches its production shape: the refusal case carries a runtime code, the model case carries none. A pair of synthetic bare errors does not satisfy this AC — that was the loophole in this ticket's first revision.
• Coverage red on today's shape, proved by mutation in both directions: reverting the mint, and dropping the code in the tenant-sync middle (assertErrorFreeIngestionSummary / getSourceErrorCode). A witness green under either mutation does not span the chain.
• Every emitted code is a literal declared in our own module, with a format-valid untrusted negative — a provider-authored KB_-shaped string must be refused. Shape alone is not provenance.
• No provider message reaches the surface. :574's message interpolates the model identifier and observed ids; the mint must carry the classification, not the text.
• KB_VECTOR_EMBED_FAILED is reachable only for shape 3, and the remaining paths to it are enumerated.

Dependencies

Unblocked by #16647 / PR #16657. The predecessor merged to dev as 9376bc81fa on 2026-08-08. This ticket consumes the now-shipped closed classification boundary by adding a source-owned generic cause and an exact trusted Node-code translation; INTERNAL_EMBED_ERROR_CODES remains unchanged. The native blocked-by edge is retained as historical dependency provenance. Raised at intake by @neo-gpt and revalidated against current origin/dev.

Out of Scope

• The classification boundary itself — PR #16657. This ticket feeds it; it does not rebuild it.
• The writer/reader/projector validator split. Three bounded-code gates exist at different trust positions and are deliberately independent; unifying them removes defence-in-depth, not duplication.
• Retry policy for model-load failures — :781's detector is in scope only as a place that already knows a cause and mints nothing. Changing when or how often it retries is not.
• #16566 (embed-stage failure on this plane), #14154 (embedder eviction root cause), #16012 (saturation retry policy), #16577 (zero-chunk materialization). Adjacent, surfaced at intake, none duplicate.

Avoided Traps

• Treating "no code" and "not in our vocabulary" as one defect. They need opposite repairs, and conflating them yields an AC that green-lights on synthetic inputs. This ticket's first revision did exactly that — the same label-class conflation this deployment family keeps producing.
• Lifting describeEmbeddingProbeFailure unchanged. It collapses every unknown cause to one string and cannot meet AC1.
• Reading a shared error code as a shared fault. A fallback says the most when the cause is known and the least when it is not.
• Surfacing the raw message. The fix is a minted code. :574's message is the clearest case for why: it carries the diagnosis and interpolated configuration.

Decision Record impact

none — applies an established boundary discipline to provider branches that predate it.

Related

#16647 (parent — the coded half, delivered by PR #16657) · PR #16657 · #16566 · #16568 · adjacent-not-duplicate: #14154, #16012, #16577.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-08T04:59:53Z; no equivalent found. A2A in-flight claim sweep: 14 most recent messages scanned, no competing [lane-claim] on this scope. Intake sweep by @neo-gpt at 2026-08-08T05:18:14Z surfaced the three adjacent tickets above.

Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2

Retrieval Hint: query_raw_memories("embed failure cause known in-process and never minted, model-not-resident bare Error vs ECONNREFUSED untrusted code, three input classes")

Retrieval Hint: the discriminating read is that TextEmbeddingService:781 already recognizes model-not-resident by message substring and acts on it — the knowledge exists and dies there.