A failing embedding write canary is never cached, so it re-runs on every healthcheck probe and cannot recover

Family anchor. This is one of three poles of the same missing primitive. See "The family" below — the fix should be a shared bounded-retry-with-reason primitive, not three local patches.

The family: one missing primitive, three different signs

Three independently-discovered failures turn out to be the same gap — no shared notion of bounded retry that records why it stopped.

The canary retries when it must not. TenantRepoSync stops retrying and never resumes. The backfill retries something that can never succeed. Each was patched-in-isolation-shaped when found alone; together they describe one primitive nobody has: retry with a bounded attempt budget, a recorded stop reason, and a guaranteed resumption condition.

Two secondary gaps fall out of the same place and are worth carrying with the family:

• No stop reason is retained. TenantRepoSync shows three of four repos with seven consecutive failures each and lastErrorCode: null — a counter that increments without recording a cause turns a one-command fix into a guessing exercise.
• Green surfaces over a dead lane. A suppressed lane reports status: completed, lastExitCode: 0, lastError: null, with a lastSuccessAt seconds old — because the sweep genuinely succeeded at attempting nothing. Silence from backoff is indistinguishable from recovery, and selfHeal recorded zero events ever against a permanently starved lane. "Required lane suppressed beyond N hours with zero successes" is a detectable condition that nothing currently detects.

Implementation of the family is declared by @neo-opus-vega following #16208.

Context

ai/scripts/diagnostics/mcpHealthcheck.mjs is the process container healthchecks run. It calls the MCP healthcheck tool and exits non-zero unless the returned status is healthy.

The Memory Core healthcheck tool performs an embedding write canary as part of that call, bounded by embeddingWriteCanaryTimeoutMs (default 30000, ai/mcp/server/memory-core/configBase.mjs:386).

HealthService.#getEmbeddingWriteCanary caches the canary result for 60s — but only on success:

if (result.status === 'healthy') {
    this.#embeddingWriteCanaryCache = {key, checkedAt: now, result};
} else {
    this.#embeddingWriteCanaryCache = null;   // failures are NOT cached
}

Problem

That asymmetry creates two regimes with very different costs, and the expensive one is self-sustaining.

Healthy: the canary runs at most once per 60s regardless of probe frequency. A 10s container interval costs ~1,440 canaries/day. Bounded and fine.

Failing: every failure nulls the cache, so the next probe runs the canary again. With a typical interval: 10s against a 30000 ms deadline, attempts overlap and accumulate — interval < timeout makes concurrent canaries reachable by construction.

The failing regime cannot exit itself. Repopulating the cache requires a healthy canary, which requires a responsive provider, which requires load to drop — but the load is now one uncached canary per probe, plus whatever overlapping attempts are still in flight. A transient slowdown latches into a permanent one.

The asymmetry is deliberate, and this supersedes its stated rationale. HealthService.mjs:909-911:

"Cache only healthy canaries so degraded probes still retry immediately, while a recent success avoids adding pressure to the interactive embedding path during diagnostics."

That reasoning holds when failures are rare and retry-worthy. Under provider saturation they are neither — and the goal the comment names, sparing the interactive embedding path, is precisely what the behaviour defeats. Whoever implements this should retire that assumption explicitly rather than silently inverting it.

Overlap is structural, not just reachable. The container probe and the in-server canary have independent lifetimes. A compose probe with timeout: 10s against a 30s canary deadline means Docker kills the probe at 10s while the server-side canary keeps running to completion. Abandoned canaries therefore stack inside the server no matter what the probe script does — so the single-flight join point has to live inside #getEmbeddingWriteCanary, not in mcpHealthcheck.mjs. Client-side guarding cannot reach this.

It is also undiagnosable from inside the affected service: the Memory Core reports degraded on its own probe timing out, while the cause is its own probe cadence contending for a provider that may be shared with sibling containers.

Evidence: two deployments, same build, opposite regimes

B makes 6x fewer probes yet spends 15x longer per call. Its maxDurationMs of 37,749 exceeds even the 30,000 ms deadline. That is ~4.3 hours/day of wall time inside liveness probes, and it is the signature of the canary running on essentially every call rather than once a minute.

The user-visible cost on B, same window — semantic recall contends with canaries for the same provider:

query_raw_memories   avg  53.7 s   max 295 s
query_summaries      avg 114.3 s   max 196 s

A memory-mining sweep during this state looks like a broken tool.

Architectural Reality

A liveness check answers "is this process able to serve?". It should be cheap, bounded, and side-effect free.

An embedding write canary answers a different and genuinely valuable question: "is the write path to the embedding provider working end to end?". Worth asking — but not at liveness frequency, and not from the probe whose failure restarts the container.

Coupling them makes the expensive question inherit the cheap question's cadence. The 60s cache was the mitigation for that, and it works exactly until the moment it is most needed: the cache is bypassed precisely when the provider is already struggling.

Note the interaction with retries in a typical compose healthcheck: sustained self-inflicted timeouts accumulate toward an unhealthy verdict, so this can escalate from degraded to a restart loop.

Fix

Ordered by how much each contributes to breaking the cycle:

1. Back off a failing canary. Failure must not retry at probe frequency. Cache the failure with a short TTL, or apply exponential backoff. This alone breaks the loop and is the minimal change.

2. Single-flight the canary, inside #getEmbeddingWriteCanary. Never let an attempt overlap its predecessor. The join point must be server-side: a probe killed at its own timeout does not stop the canary it started, so client-side guarding cannot prevent server-side stacking.

   A proven shape already exists in this codebase — PR #16209 shipped exactly this: a promise assigned synchronously, before any await, so concurrent callers join rather than race, cleared on settle so failures retry fresh and a rejection is never cached. RecorderService.ensureStore() on dev is lift-ready, and it came with a measured falsifier worth copying: 32 parallel calls produced 32 opens before the change and 1 after. The equivalent here is N concurrent healthchecks under a saturated provider producing 1 canary attempt, not N.

3. Decouple the canary from routine liveness. Default healthcheck to the cheap path, run the canary on its own slow cadence with the last result reported. Subsumes 1 and 2, and is the right end state — a liveness probe should perform no inference at all.

4. Report the reason. A degraded verdict should distinguish "provider saturated, backing off" from "provider unreachable". Today those are indistinguishable, and only one is self-inflicted.

5. Document a sane container probe interval. 10s is oversampling for a probe permitted 30s. Minutes is the right order for this class of service. Note this is a mitigation: it lengthens the loop without breaking it, so it does not substitute for 1 and 2.

Out of scope

• #13435 (decoupling the in-container healthcheck from the gitlab-pat user-token gate) — adjacent, different concern.

Acceptance criteria

1. A container liveness probe issues no embedding request; verified by tool metrics showing healthcheck calls with no corresponding embedding-provider activity.
2. Under a saturated provider and sustained probing, canary attempts decrease rather than running once per probe, and no attempt overlaps another.
3. A deployment that has entered the degraded state recovers without operator intervention once the provider becomes responsive.
4. The embedding write canary remains available and is exercised on an explicitly configured cadence independent of the liveness interval.
5. A degraded verdict caused by canary timeout carries its reason, distinguishing provider saturation from provider failure.

Contract Ledger (claimer-authored section — @neo-opus-vega, per intake-derived-ledger convention; body owner may reshape)

Live-evidence residuals (AC-mapped, owner @neo-opus-vega)

• AC 1 + AC 2 (live) — saturated-plane receipt: probes-run-zero-inference and attempts-DECREASE observed on a running deployment at the active rebuild-run boundary.
• AC 3 (live) — autonomous recovery observed once the provider drains, no operator action.
• AC 4 (docs) — probe-interval-vs-cadence guidance: landed on the cadence config leaf JSDoc (probes free to differ; N probes per window cost zero inference).