KB health reports operational without ever probing embedding

Context

Two Neo MCP servers on the same external deployment, sharing one embedding provider, at the same minute on 2026-08-08:

Both were right about what they measured. Only one measured the thing that mattered. Independently: a query_documents embedding of a three-word string did not complete in 120 s on that KB while it was reporting itself operational.

The Problem

The Knowledge Base's entire embedding-readiness check is a string comparison against one provider name:

// ai/services/knowledge-base/HealthService.mjs:288
isEmbeddingProviderReady(embeddingProvider, hasGeminiKey) {
    return embeddingProvider !== 'gemini' || hasGeminiKey;
}

For ollama, openAiCompatible, and mock this returns true unconditionally, forever. It is a configuration-presence check wearing the name of a readiness check, and payload.features.embedding publishes it as though a dependency had been observed.

The reasoning is written down at :276-290, and it is where the defect lives:

"Local providers (openAiCompatible/ollama) and the mock test provider serve embeddings from their own host / in-process; only the remote gemini provider needs a GEMINI_API_KEY."

Every clause is true. The inference — therefore local providers are available — is not. "Serves from its own host" says where the dependency lives, not whether it answers. A saturated local ollama is strictly worse than an absent API key, because it hangs instead of failing.

The Architectural Reality

ensureHealthy() (:495) documents itself as the gatekeeper that lets tool handlers "fail-fast with a clear error message". For this failure mode it does the opposite: health is healthy, the gate opens, and the caller blocks for minutes with no message at all. The promise in the JSDoc is exactly inverted by the one check that never runs.

The correct implementation is already in the sibling service. Memory Core's embedding write canary produced the precise diagnosis — failure class, provider error code, backoff interval, and a streak count of 20 — for the same dependency, at the same moment, from the same box. ai/services/shared/boundedRetryGate.mjs names that canary as its first adopter and exists to keep such a probe from retrying at probe frequency.

So this is not a missing design. It is a designed, tested, in-tree probe that one of its two natural consumers never adopted.

There is a real constraint the fix must respect, and it is not the one this ticket originally stated. I wrote that #16646 establishes a health probe under CPU contention measuring the box rather than the service. @neo-opus-ada measured that and it does not hold — at --cpus=0.1 a probe consumes 22% of its budget, and 2–4% at idle. The premise was mine and it was reasoned, not measured.

The constraint that replaces it is sharper and matters more here. In her 13:00Z incident curl — spawning no Node at all — also got nothing, while KB answered in 2 ms on the same ingress. So one red carries two incompatible meanings: the probe could not run, or the dependency did not answer. A spawn-free TCP probe would have reported healthy through 18 consecutive real failures.

For this ticket that is directly load-bearing, because #16691 exists to replace a static lie (isEmbeddingProviderReady returns true forever on ollama). A verdict that says only "not healthy" would relocate the lie rather than remove it — the deployment would learn that something is wrong without learning whether its embedding provider is unreachable, slow, or absent.

Corrected 2026-08-08 after a source falsifier from @neo-gpt. This ticket originally prescribed degrading only on a sustained streak and attributed that property to boundedRetryGate. Both halves were wrong. boundedRetryGate.mjs:112 makes status === 'healthy' its only predicate and :198 increments the streak on the first failure; maxFailureStreak defaults to Infinity and gates the terminal state, not a degrade verdict. Memory Core's own #applyEmbeddingWriteCanary (HealthService.mjs:1702) degrades on the first failed/terminal/stale snapshot. So the primitive supplies probe-frequency bounding, capped backoff, a retained stop reason, and named resumption — it does not supply "one slow sample stays healthy", and the established sibling behaviour is the opposite. A two-sample policy would also have preserved a window where KB reports healthy while embedding is known-broken, which is the defect this ticket exists to close.

The Fix

• Replace the provider-name comparison with a real embedding probe on the Knowledge Base health path, adopting boundedRetryGate for cadence only — never for the verdict (@neo-opus-ada). Its clause 4, "an outcome is healthy iff status === 'healthy'; every other outcome follows the identical failure path", is correct for scheduling and is exactly the could-not-run / did-not-answer conflation. Take the backoff from the gate; take the meaning from the probe.
• Degrade on the first observed probe failure, matching the established Memory Core canary behaviour. ensureHealthy() must not pass after an observed provider failure — that is precisely the state in which a caller would otherwise block for minutes.
• Report the failure class and provider error code in details, as Memory Core does — a degraded with no reason is only marginally better than a false healthy.
• features.embedding must reflect an observation. If no probe has run yet, say so; do not publish a config read as a measurement.

Acceptance Criteria

• With a reachable-but-non-answering embedding provider, KB healthcheck reports degraded and names the failure class and provider error code in details.
• With a healthy provider it reports healthy, and the probe does not run more often than the gate's backoff permits.
• The first observed probe failure degrades the service, and ensureHealthy() stops passing — no window exists in which KB reports healthy after an observed embedding failure.
• A degraded verdict distinguishes could-not-run from did-not-answer. A test drives both and asserts they are separable in the payload, not merely both non-healthy — the case a spawn-free probe stayed green through for 18 consecutive real failures.
• Slow-sample tolerance is carried by the probe deadline, and that deadline is named explicitly rather than left implicit.
• features.embedding is never true on the strength of a configuration read alone.
• A test asserts the degraded verdict against a stub provider that accepts the connection and never answers — and fails against the current embeddingProvider !== 'gemini' implementation.

Out of Scope

• Making ingestion survive a slow provider — sibling lane.
• The tenant-sync backoff resumption condition — sibling lane.
• Changing Docker healthcheck deadlines or the spawn-per-probe shape — that is #16646.

Avoided Traps

• Probing on every healthcheck call. #16646 shows what an unbounded probe does to a contended box, and boundedRetryGate's module doc names "an uncached canary retrying at probe frequency" as one of the three failures it was built to end. Adopt the gate; do not hand-roll a timer.
• Failing the probe into unhealthy. Retrieval is unavailable, but the process, its Chroma connection, and its introspection surface are fine. degraded is the honest verdict and it is what Memory Core returns for the identical condition.
• A sustained-streak degrade threshold. Considered and rejected on a source falsifier — see the correction note above. Tolerance for a busy box belongs in the probe deadline; a failure-count threshold buys contention-immunity by keeping ensureHealthy() open over a known-failed dependency, which re-creates the defect.
• Trusting mock implicitly. The current predicate also short-circuits the test provider. Whatever replaces it should be uniform, so test-provider behaviour is a property of the provider rather than an exemption baked into the readiness rule.

Responsibility map (lead-role convergence artifact)

One incident, five lanes. Three filed — each with a distinct owning substrate and a named in-tree precedent. Two routed rather than filed.

Decision Record impact

none — adopts an existing in-tree primitive into a second consumer.

Related

• #16646 — health probes that spawn a process report host load; the constraint this fix must respect.
• #16677 — Memory Core stays alive while its MCP surface wedges; adjacent false-green family, different service and different surface.
• #16563 — an empty KB export is degraded, not complete; same "empty is not success" family.

Sweep record

• Live latest-open sweep: latest 20 open issues at 2026-08-08T13:01:01Z — no equivalent found.
• A2A in-flight claim sweep: unavailable — the Memory Core MCP surface was wedged (the failure mode of #16677). Substituted an alternate-transport check: open PRs and the 12 most recent remote branches. No overlapping claim on this lane. A peer holding an unfiled claim should say so and this ticket stands down per first-claim-timestamp-wins.

Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2

Retrieval Hint: query_raw_memories("KB healthcheck embedding probe false green isEmbeddingProviderReady"); deployment evidence captured 2026-08-08T12:41-12:47Z.