LearnNewsExamplesServices
Frontmatter
id17066
titleKB and MC gate their diagnostic reads behind full operability, so an embedding outage blinds the surface needed to diagnose it
stateOpen
labels
bugaimodel-experienceagent-os
assignees[]
createdAt10:17 PM
updatedAt10:17 PM
githubUrlhttps://github.com/neomjs/neo/issues/17066
authorneo-opus-vega
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]

KB and MC gate their diagnostic reads behind full operability, so an embedding outage blinds the surface needed to diagnose it

Open Backlog/active-chunk-15 bugaimodel-experienceagent-os
neo-opus-vega
neo-opus-vega commented on 10:17 PM

Context

During a live external-plane embedding outage on 2026-08-13, every attempt to characterise the incident through the Knowledge Base MCP surface returned the same refusal:

Cannot execute list_documents: Knowledge Base is not fully operational:
  - Connected to the orchestrator-managed ChromaDB instance
  - Knowledge Base embedding probe failed: provider-timeout:OPENAI_COMPATIBLE_REQUEST_TIMEOUT
    — backing off 600000ms (streak 39, deadline 300000ms)

The same guard blocked query_documents and get_deployment_state_snapshot. Memory Core refused query_raw_memories on its own embedding canary.

Everything needed to answer "which repos are in the corpus and what stalled" was gated by the failure being investigated. Only healthcheck and the orchestrator bridge (inspect_deployment) survived — the incident was ultimately root-caused entirely through the bridge, because the servers' own read surface was closed.

The Problem

The operability gate is correct for operations that require the failed dependency. It is applied far more broadly than that:

  • list_documents enumerates rows. It does not embed.
  • get_deployment_state_snapshot reads a JSON file written by the orchestrator. It does not embed.
  • Counting collection contents does not embed.

Conflating "the embedding provider is down" with "this server cannot answer questions" removes observability precisely when observability matters most. The operator loses the ability to answer basic questions — which tenant repos are represented, how many items each contributed, when the corpus last changed — during the only window in which those answers are urgent.

This has a real cost beyond inconvenience: an outage that blinds its own diagnostics extends by however long it takes to find an alternative surface. On this plane the corpus question ("did any other repo get in?") could not be answered directly at all; it had to be inferred from an orchestrator-side lane report.

Architectural Reality

  • The gate is a coarse status !== healthy check rather than a per-tool dependency declaration.
  • Tools already differ sharply in what they need: embedding-dependent (semantic query, ingest), storage-dependent (list, count, snapshot read), and neither (handbook, healthcheck).
  • The degraded state is not binary in practice — on the observed plane Chroma was connected and readable the entire time; only the embedding provider was failing.

The Fix (shape)

  1. Declare each tool's actual dependencies rather than gating on aggregate health. A tool blocks only when a dependency it genuinely uses is unavailable.
  2. Where a tool can answer partially, answer partially and say so — a document list with a banner that semantic ranking is unavailable is strictly more useful than a refusal.
  3. Preserve the refusal for operations that would produce wrong results if run degraded (semantic query against a dead embedder must not silently fall back).
  4. Treat "can this surface still be diagnosed while degraded?" as a design requirement for MCP read tools, not an afterthought.

Acceptance Criteria

  • With the embedding provider failing and storage healthy, list_documents and get_deployment_state_snapshot return results.
  • Semantic-query tools still refuse under the same conditions, with the reason naming the specific unavailable dependency.
  • Per-tool dependency declarations exist and are the input to the gate; no tool is gated by a dependency it does not use.
  • A partially-degraded response is distinguishable from a fully-healthy one by the caller.
  • Regression: a fixture with a dead embedder and live storage asserts the read/refuse split.

Out of Scope

  • The embedding outage itself (#17062 · #17063).
  • Adding new MCP tools — this narrows existing gates and adds no surface (see the tool-cap discipline).

Avoided Traps

  • "Degraded means degraded." Aggregate health is a summary for humans; it is the wrong input to a per-tool authorization decision.
  • "Callers might misread partial data." Addressed by labelling the response, not by withholding it.
  • "The bridge covered it." Only because an orchestrator happened to be present and reachable. A KB deployed without one would have been fully opaque.

Related