Context
During a live external-plane embedding outage on 2026-08-13, every attempt to characterise the incident through the Knowledge Base MCP surface returned the same refusal:
Cannot execute list_documents: Knowledge Base is not fully operational:
- Connected to the orchestrator-managed ChromaDB instance
- Knowledge Base embedding probe failed: provider-timeout:OPENAI_COMPATIBLE_REQUEST_TIMEOUT
— backing off 600000ms (streak 39, deadline 300000ms)The same guard blocked query_documents and get_deployment_state_snapshot. Memory Core refused query_raw_memories on its own embedding canary.
Everything needed to answer "which repos are in the corpus and what stalled" was gated by the failure being investigated. Only healthcheck and the orchestrator bridge (inspect_deployment) survived — the incident was ultimately root-caused entirely through the bridge, because the servers' own read surface was closed.
The Problem
The operability gate is correct for operations that require the failed dependency. It is applied far more broadly than that:
list_documents enumerates rows. It does not embed.
get_deployment_state_snapshot reads a JSON file written by the orchestrator. It does not embed.
- Counting collection contents does not embed.
Conflating "the embedding provider is down" with "this server cannot answer questions" removes observability precisely when observability matters most. The operator loses the ability to answer basic questions — which tenant repos are represented, how many items each contributed, when the corpus last changed — during the only window in which those answers are urgent.
This has a real cost beyond inconvenience: an outage that blinds its own diagnostics extends by however long it takes to find an alternative surface. On this plane the corpus question ("did any other repo get in?") could not be answered directly at all; it had to be inferred from an orchestrator-side lane report.
Architectural Reality
- The gate is a coarse
status !== healthy check rather than a per-tool dependency declaration.
- Tools already differ sharply in what they need: embedding-dependent (semantic query, ingest), storage-dependent (list, count, snapshot read), and neither (handbook, healthcheck).
- The degraded state is not binary in practice — on the observed plane Chroma was connected and readable the entire time; only the embedding provider was failing.
The Fix (shape)
- Declare each tool's actual dependencies rather than gating on aggregate health. A tool blocks only when a dependency it genuinely uses is unavailable.
- Where a tool can answer partially, answer partially and say so — a document list with a banner that semantic ranking is unavailable is strictly more useful than a refusal.
- Preserve the refusal for operations that would produce wrong results if run degraded (semantic query against a dead embedder must not silently fall back).
- Treat "can this surface still be diagnosed while degraded?" as a design requirement for MCP read tools, not an afterthought.
Acceptance Criteria
Out of Scope
- The embedding outage itself (#17062 · #17063).
- Adding new MCP tools — this narrows existing gates and adds no surface (see the tool-cap discipline).
Avoided Traps
- "Degraded means degraded." Aggregate health is a summary for humans; it is the wrong input to a per-tool authorization decision.
- "Callers might misread partial data." Addressed by labelling the response, not by withholding it.
- "The bridge covered it." Only because an orchestrator happened to be present and reachable. A KB deployed without one would have been fully opaque.
Related
Context
During a live external-plane embedding outage on 2026-08-13, every attempt to characterise the incident through the Knowledge Base MCP surface returned the same refusal:
Cannot execute list_documents: Knowledge Base is not fully operational: - Connected to the orchestrator-managed ChromaDB instance - Knowledge Base embedding probe failed: provider-timeout:OPENAI_COMPATIBLE_REQUEST_TIMEOUT — backing off 600000ms (streak 39, deadline 300000ms)The same guard blocked
query_documentsandget_deployment_state_snapshot. Memory Core refusedquery_raw_memorieson its own embedding canary.Everything needed to answer "which repos are in the corpus and what stalled" was gated by the failure being investigated. Only
healthcheckand the orchestrator bridge (inspect_deployment) survived — the incident was ultimately root-caused entirely through the bridge, because the servers' own read surface was closed.The Problem
The operability gate is correct for operations that require the failed dependency. It is applied far more broadly than that:
list_documentsenumerates rows. It does not embed.get_deployment_state_snapshotreads a JSON file written by the orchestrator. It does not embed.Conflating "the embedding provider is down" with "this server cannot answer questions" removes observability precisely when observability matters most. The operator loses the ability to answer basic questions — which tenant repos are represented, how many items each contributed, when the corpus last changed — during the only window in which those answers are urgent.
This has a real cost beyond inconvenience: an outage that blinds its own diagnostics extends by however long it takes to find an alternative surface. On this plane the corpus question ("did any other repo get in?") could not be answered directly at all; it had to be inferred from an orchestrator-side lane report.
Architectural Reality
status !== healthycheck rather than a per-tool dependency declaration.The Fix (shape)
Acceptance Criteria
list_documentsandget_deployment_state_snapshotreturn results.Out of Scope
Avoided Traps
Related