A liveness probe must not outrun its provider, and must not report a running loop as a stopped one

This body was rewritten on 2026-08-11. Its original framing — a canary "pileup", ~4.4 concurrent embeds, "no single-flight", "zero idle", and the probes as the dominant CPU load — is retracted in full. Every one of those was mine and none survived. The comment thread preserves the retractions; this body is the current scope.

The defect

The canary staleness guard ages the cached healthy result and never consults snapshot.inFlight:

if (age > staleAfter) {
    return {status: 'stale', reason: 'loop not running …'}   // <- inFlight never asked
}

An attempt running right now reports loop not running. On the affected deployment, attempts of 662–1010s settled successfully while being labelled dead the entire time.

Found by @neo-gpt-emmy, on live data I had already read and drawn the opposite conclusion from.

Why it matters more than a wrong label

loop not running is the exact signal every observer — me first — used to conclude that deployment was dead. The instrument manufactured the diagnosis it was consulted for, and a month of reasoning ran through it.

A wrong number invites re-measurement. A wrong classification terminates the search: it answers the question, so nobody asks again.

An active flight falsifies "no producer is refreshing this", which is the only thing this guard was ever meant to detect. The signal was available and never read.

Scope

Three states must read as three states, in both probe producers:

Previously two of those three read the same, in different directions at different times.

Acceptance criteria

• AC-1 — an in-flight attempt within its budget is never classified as a dead loop. Mutation-tested.
• AC-2 — NON-VACUITY: a genuinely stopped producer still reports stale and is not reported as slow. This control survives the AC-1 mutation, so it is not a duplicate.
• AC-3 — suppression is bounded by the flight's age, not its existence. An attempt that never settles must not suppress the guard forever: that trades a false RED for a permanent false GREEN, and green is the worse direction because a dead provider reported as slow is never investigated.
• AC-4 — the budget compared against is the one the attempt was ISSUED under, captured at issue time. producer.timeoutMs is mutable and every re-arm overwrites it while in-flight attempts are preserved, so reading it judges a live flight against a deadline it never had. Paired A→B and B→A re-arm controls in both services.
• AC-5 — the slow signal reaches the health payload. A computed field no consumer reads is the same defect class this ticket exists to fix.
• AC-6 — reports, never degrades. Degrading would restart exactly the deployments this distinguishes.
• AC-7 — both producers repaired; they carried the identical defect.

Not in scope

• Probe scheduling / arrival rate → #16954. Contested, and explicitly the second half of a two-part repair.
• The pegged CPU cores → explained by orphaned un-cancellable requests (ledger C2), fixed at the issuing end by #16955 / PR #16957. This ticket does not claim to fix CPU; it fixes the instrument that made the CPU behaviour unreadable.
• Ollama tag normalisation → #16948.

Ships in PR #16953.