LearnNewsExamplesConnect
Frontmatter
assignees[]
createdAtAug 11, 2026, 10:39 AM
updatedAtAug 11, 2026, 12:58 PM
githubUrlhttps://github.com/neomjs/neo/issues/16951
authorneo-opus-grace
commentsCount3
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 11, 2026, 12:58 PM

A liveness probe must not outrun its provider, and must not report a running loop as a stopped one

neo-opus-grace
neo-opus-grace commented on Aug 11, 2026, 10:39 AM

This body was rewritten on 2026-08-11. Its original framing — a canary "pileup", ~4.4 concurrent embeds, "no single-flight", "zero idle", and the probes as the dominant CPU load — is retracted in full. Every one of those was mine and none survived. The comment thread preserves the retractions; this body is the current scope.

The defect

The canary staleness guard ages the cached healthy result and never consults snapshot.inFlight:

if (age > staleAfter) {
    return {status: 'stale', reason: 'loop not running …'}   // <- inFlight never asked
}

An attempt running right now reports loop not running. On the affected deployment, attempts of 662–1010s settled successfully while being labelled dead the entire time.

Found by @neo-gpt-emmy, on live data I had already read and drawn the opposite conclusion from.

Why it matters more than a wrong label

loop not running is the exact signal every observer — me first — used to conclude that deployment was dead. The instrument manufactured the diagnosis it was consulted for, and a month of reasoning ran through it.

A wrong number invites re-measurement. A wrong classification terminates the search: it answers the question, so nobody asks again.

An active flight falsifies "no producer is refreshing this", which is the only thing this guard was ever meant to detect. The signal was available and never read.

Scope

Three states must read as three states, in both probe producers:

condition reading
in flight, within its issued budget healthy + a named slow detail
in flight, past its issued budget stale — the deadline that should have ended it did not fire
nothing in flight, cache aged out stale — a genuinely dead loop

Previously two of those three read the same, in different directions at different times.

Acceptance criteria

  • AC-1 — an in-flight attempt within its budget is never classified as a dead loop. Mutation-tested.
  • AC-2 — NON-VACUITY: a genuinely stopped producer still reports stale and is not reported as slow. This control survives the AC-1 mutation, so it is not a duplicate.
  • AC-3 — suppression is bounded by the flight's age, not its existence. An attempt that never settles must not suppress the guard forever: that trades a false RED for a permanent false GREEN, and green is the worse direction because a dead provider reported as slow is never investigated.
  • AC-4 — the budget compared against is the one the attempt was ISSUED under, captured at issue time. producer.timeoutMs is mutable and every re-arm overwrites it while in-flight attempts are preserved, so reading it judges a live flight against a deadline it never had. Paired A→B and B→A re-arm controls in both services.
  • AC-5 — the slow signal reaches the health payload. A computed field no consumer reads is the same defect class this ticket exists to fix.
  • AC-6 — reports, never degrades. Degrading would restart exactly the deployments this distinguishes.
  • AC-7 — both producers repaired; they carried the identical defect.

Not in scope

  • Probe scheduling / arrival rate → #16954. Contested, and explicitly the second half of a two-part repair.
  • The pegged CPU cores → explained by orphaned un-cancellable requests (ledger C2), fixed at the issuing end by #16955 / PR #16957. This ticket does not claim to fix CPU; it fixes the instrument that made the CPU behaviour unreadable.
  • Ollama tag normalisation → #16948.

Ships in PR #16953.

tobiu referenced in commit 82470ab - "fix(memory-core): a running loop was reported as a stopped one (#16951) (#16953) on Aug 11, 2026, 12:58 PM
tobiu closed this issue on Aug 11, 2026, 12:58 PM
tobiu referenced in commit d22241b - "fix(memory-core): a liveness probe must not be able to occupy its own subject (#16951) on Aug 21, 2026, 2:32 AM
tobiu referenced in commit 18a253e - "fix(memory-core): one duty-cycle policy, both probe producers, honest about what it bounds (#16951) on Aug 21, 2026, 2:32 AM
tobiu referenced in commit 57aa753 - "chore(memory-core): record the two duty-cycle leaves in the parity snapshot (#16951) on Aug 21, 2026, 2:32 AM
dawesi referenced in commit 8e2f123 - "fix(memory-core): a running loop was reported as a stopped one (#16951) (#16953) on Aug 25, 2026, 5:41 PM