LearnNewsExamplesServices
Frontmatter
id17296
titleThe embedding lane reports its shape but never its model identity
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtAug 17, 2026, 1:56 PM
updatedAtAug 18, 2026, 12:33 PM
githubUrlhttps://github.com/neomjs/neo/issues/17296
authorneo-opus-vega
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 18, 2026, 12:33 PM

The embedding lane reports its shape but never its model identity

Closed Backlog/active-chunk-17 bugaiarchitecture
neo-opus-vega
neo-opus-vega commented on Aug 17, 2026, 1:56 PM

The embedding lane reports its shape but never its model identity, so a wrong model serves silently

Context

A production plane ran the wrong embedding model across two consecutive deploys without any surface reporting it. The deployment intended qwen3-embedding-0.6b; the container served Qwen3-Embedding-8B-Q4_K_M. Both deploys completed successfully, every container reported healthy, and the only symptom was that embeddings timed out — read as a performance problem, not a configuration one.

Proving it required reading two literals out of a container log tail by hand plus a live geometry probe. The plane had the facts and never volunteered them.

The Problem

Neo has the check. TextEmbeddingService raises, with the right words:

LM Studio embedding model '<configured>' is not resident under its configured identifier; observed=<ids>

tagged residencyDisposition = EMBEDDING_RESIDENCY_NEVER_RESIDENT. It cannot fire on a containerized plane, because #shouldAssertOpenAiCompatibleEmbeddingContext() (ai/services/memory-core/TextEmbeddingService.mjs:971-995) gates it behind an LM Studio port comparison:

if (aiConfig.orchestrator.lms.enabled !== true) return false;
…
return endpointUrl.port === lmsPort;

Its own JSDoc states the scope honestly: "OpenAI-compatible covers LM Studio, Ollama's compatibility surface, llama.cpp, vLLM, and CI fixture servers. The lms ps loaded-context probe is valid only for the LM Studio CLI lane." The gate is correct for the probe it guards — lms ps is genuinely LM-Studio-only. The consequence is that identity verification exists for the developer laptop and not for the deployment, which is the inverse of where it is needed.

The bridge layer has the complementary half of the same gap. orchestrator.deploymentStateBridge.providerResidencyServiceKeys defaults to ['local-model', 'model'] (ai/configBase.mjs:1427), so residency — which models are loaded — attaches only to the Ollama containers. The embedding service gets providerLaneShape instead, a deliberately separate key (the sibling leaf's JSDoc says so explicitly).

The observed asymmetry on that plane, at the same instant:

service reports says
local-model (chat, idle) providerResidency "missing gemma4:26b … pull missing configured model(s)" — a defect, in plain words
embedding-model (serving the WRONG model) providerLaneShape only parallelism 4, contextTokensPerSlot 32768 — geometry, no identity, providerResidency: null

The lane that merely serves chat volunteers its model identity. The lane whose model was silently wrong reports only its slot geometry. The asymmetry is inverted relative to risk.

The Architectural Reality

  • ai/services/memory-core/TextEmbeddingService.mjs:971-995 — the LMS-port gate.
  • …:1036-1060 — the identity check itself, including the observed= diagnostic and the EMBEDDING_RESIDENCY_NEVER_RESIDENT disposition. Good code, unreachable on llama.cpp.
  • ai/configBase.mjs:1427providerResidencyServiceKeys: leaf(['local-model', 'model'], …).
  • services/DeploymentStateBridgeService.mjs:1077isProviderResidencyServiceKey(), and at :1082-1095 a JSDoc that already documents a previous incident where a zero intersection left both residency fields null for a deployment's life "so the reader concluded the provider was unobservable rather than unasked". Same class, different key.
  • llama.cpp's OpenAI-compatible surface serves GET /v1/models, and the compose already sets LLAMA_ARG_ALIAS=qwen3-embedding-0.6b — the served identifier the configured value should be compared against. The probe does not need lms ps.

The Fix

Identity verification becomes provider-shaped rather than vendor-shaped: any OpenAI-compatible endpoint can be asked GET /v1/models and its answer compared against the configured openAiCompatible.embeddingModel. The existing error text, disposition and preflight wiring are reused unchanged — only the gate widens. Whether the mismatch refuses at boot or degrades health is the design call this ticket leaves to its implementer; the plane must at minimum state it.

Second half: the embedding service becomes residency-observable at the bridge, so the mismatch reaches inspect_deployment the way Ollama's missing model already does.

Acceptance Criteria

  • An OpenAI-compatible endpoint that is not LM Studio has its served model identity compared against the configured embeddingModel, and a mismatch is reported with the served identifier named — witnessed against a llama.cpp-shaped /v1/models response.
  • The LM Studio lane keeps its current behaviour, including the lms ps context probe, which remains correctly vendor-gated.
  • A mismatch is visible on a surface an operator or agent reads without shell access — inspect_deployment and/or the MCP healthcheck — not only as a thrown error at embed time.
  • Red-proof against the real incident: a lane configured for qwen3-embedding-0.6b while serving Qwen3-Embedding-8B-Q4_K_M reports the mismatch. The pre-fix run must be silent.
  • Unset or unreadable model lists degrade to unknown, never to a false match — an unreachable endpoint must not read as identity-confirmed.

Out of Scope

The lms ps context-length probe, which is legitimately LM-Studio-only · Ollama residency, which already works · model pulling or eviction · the deployment defect that surfaced this · #17063's sha256 healthcheck cost, adjacent but a different question.

Avoided Traps

  • Widening the lms ps probe to non-LM-Studio providers. It is vendor-specific for real reasons; the fix is a second, provider-shaped probe, not a stretched one.
  • Reading the compose file to learn the served model. The file is the intent; the incident was precisely that intent and reality diverged. Only the running endpoint can answer.
  • Assuming an alias equals a filename. LLAMA_ARG_ALIAS is the served identifier and the correct comparison target; the .gguf path is not what /v1/models returns.

Related

#17069 (closed — provider-lane shape drift; this is its identity-shaped sibling, and its closing argument that "the runtime accepts whatever shape it is given and reports healthy" applies verbatim to identity) · #17063 · #16948 (Ollama :latest literal compare — same family, other provider) · #16706 · #17295.

Live latest-open sweep: latest 20 open checked 2026-08-17T11:54Z, plus a state:all keyword sweep on four phrasings; nearest neighbours #17063 / #16948 / #17018, no equivalent. A2A herd-window sweep clean.

Origin Session ID: c992afd0-2e26-410e-b460-b480ccd0a240

Retrieval Hint: query_raw_memories("embedding model identity unverified llama.cpp lms port gate providerResidencyServiceKeys")

tobiu referenced in commit fa8c3ba - "feat(ai): the embedding lane states its model identity at embed time and on the deployment snapshot (#17296) (#17325) on Aug 18, 2026, 12:33 PM
tobiu closed this issue on Aug 18, 2026, 12:33 PM