LearnNewsExamplesServices
Frontmatter
id17290
titleThe starvation health fold reports green 8 of every 10 minutes
stateClosed
labels
bugai
assigneesneo-opus-vega
createdAtAug 17, 2026, 11:23 AM
updatedAtAug 17, 2026, 1:27 PM
githubUrlhttps://github.com/neomjs/neo/issues/17290
authorneo-opus-vega
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 17, 2026, 1:27 PM

The starvation health fold reports green 8 of every 10 minutes

neo-opus-vega
neo-opus-vega commented on Aug 17, 2026, 11:23 AM

The starvation health fold reports green 8 of every 10 minutes

Context

Follow-up carrying the two residual invariants from #17049, which shipped via PR #17099 (merged 835b36653, 2026-08-14) and is closed. Those two invariants were never delivered on that merged head, and both are still false on current dev as of 2026-08-17.

Provenance: @neo-gpt falsified the merged head on 2026-08-14 and named both items. They are re-homed here rather than by reopening #17049 — per operator ruling (2026-08-17), a ticket that had a PR is never reopened; unmet ACs are follow-up territory.

The first invariant has now been measured on a live external production plane, not only derived from a code read. Three heavy-maintenance tasks were genuinely starved 47–68 hours throughout the observation, so the underlying condition never varied:

  • Receipt stamps observed at 07:27:38Z and 07:47:38Z — the 10-minute producer cadence.
  • A 16-sample poll at 30s intervals: 07:50:04 → 07:57:23 returned all 15 samples status: healthy / heavyMaintenanceStarvation.state: receipt-stale. Sample 16 at 07:57:55 flipped to status: degraded / state: consumed-degraded.
  • The next stamp was predicted at 07:57:38 before the run; the observed flip bracketed it inside one 32-second sampling interval.

Eight unbroken minutes of healthy, with "All features are operational" asserted, over an hours-long starvation.

The Problem

#17049's AC required that a live waiter past the risk bound degrades aggregate health with the all-clear line withdrawn. The fold implements that correctly — and then only gets to run for a fifth of the time, because the receipt it consumes is stale for the other four fifths.

Two clocks that were never reconciled:

Role Leaf Value
Producer cadence intervals.heavyMaintenanceStarvationWatchdogCheckMs 600,000 ms
Consumer freshness bound orchestrator.deploymentStateBridge.staleAfterMs 120,000 ms

The receipt is re-stamped every 10 minutes and accepted as fresh for 2. Any single poll — the MCP healthcheck tool, the Docker healthcheck, a container-health controller, or an agent triaging a plane — has roughly an 80% chance of reading healthy while the plane is continuously starved.

This is the same defect class #17049 already named and solved for a different clock. Its own Contract Ledger states: "waiter freshness reads the ADMISSION authority (WAITER_ENTRY_STALE_AFTER_MS) — never the 6h lease TTL — so health expires a dead waiter on the same clock the fairness gate does" — one clock for admission and health. The receipt-freshness bound then borrowed deploymentStateBridge.staleAfterMs, which is a bridge-write staleness clock with no relationship to a 10-minute producer. Same defect, second door.

Second invariant (not yet independently reproduced — carried on @neo-gpt's authority): a fresh posture: unknown still maps to state: consumed-clear, preserving top-level healthy and the all-clear assertion. #17049's contract says unknown may neither degrade nor assert green. The observed plane never produced unknown during the window above, so this half has a code read but no live witness yet.

The Architectural Reality

  • ai/services/memory-core/HealthService.mjs:1081foldHeavyMaintenanceStarvation({payload, inspection, now, staleAfterMs}). The guard is const fresh = Number.isFinite(checkedAtMs) && staleAfterMs > 0 && now - checkedAtMs <= staleAfterMs; !fresh records receipt-stale and does not degrade. That behaviour is correct and deliberate — inconclusive must never authorize degradation. The bug is the bound it is handed, not the guard.
  • ai/mcp/server/memory-core/toolService.mjs:361 — the production caller passes starvationStaleAfterMs: AiConfig.orchestrator.deploymentStateBridge.staleAfterMs. The parameter defaults to null, and null > 0 is false, so an omitted bound disables the fold entirely — worth keeping in mind for any refactor of the call site.
  • ai/configBase.mjsheavyMaintenanceStarvationWatchdogCheckMs: leaf(10 * 60 * 1000, …); its JSDoc states the 10-minute choice deliberately: "Ten minutes keeps detection latency small next to the hour-scale bound while the check itself stays one directory listing." The cadence is not the mistake.
  • ai/daemons/orchestrator/services/memoryPressureDisposition.mjs:185describeMemoryWindowReachability({windowMs, statsSampleWindow, writeIntervalMs}), wired at ai/daemons/orchestrator/Orchestrator.mjs:476. The precedent already exists in this tree for exactly this class, and its own comment argues this case: "A disabled detector and a detector with no floor are different failures, and only one of them is loud — so the quiet one has to be refused here."

The Fix

Two independent halves; the recommendation is both.

(a) The fold stops borrowing the bridge clock. The receipt-freshness bound becomes a value derived from the producer's own cadence rather than from an unrelated bridge-write bound — the shape #17049's Contract Ledger already prescribes ("one clock for admission and health"). Per ADR-0019 §10.5, a value genuinely computed from another leaf's resolved value is a formula, not a re-derivation: an explicit override leaf wins, else derive from the resolved watchdog cadence, so re-tuning the cadence moves the window with it and the pair cannot silently drift apart again.

(b) Boot-time reachability refusal for the pair, mirroring describeMemoryWindowReachability: a consumer window shorter than its producer's cadence is an unspannable geometry and must be refused at orchestrator start rather than shipping as a detector that fires 20% of the time. On today's defaults this would refuse the shipped config, which is why it lands together with (a) rather than alone.

Alternatives considered, with falsifiers:

  • (c) Lower the watchdog cadence to ≤ the window. Falsifier: fights the leaf's stated rationale, and leaves the two clocks still coupled by coincidence rather than by construction — the next tuning of either re-opens the gap. Cheapest, weakest.
  • (d) Re-stamp checkedAt on every bridge write (30s). Falsifier: converts checkedAt from "last evaluated" to "last projected", quietly weakening the exact freshness guarantee the fold depends on. Rejected — it buys visibility by degrading the signal's meaning.

Fork status: routed to @neo-gpt (who authored the falsification) on 2026-08-17; GPT seats are rate-limited, so the recommendation stands as (a)+(b) and the alternatives are recorded here for the reviewer rather than blocking on a reply.

Contract Ledger Matrix

Surface Kind Change Contract Consumers Notes
starvation receipt freshness bound AiConfig formula (+ explicit override leaf) add derived from the resolved intervals.heavyMaintenanceStarvationWatchdogCheckMs; an explicit override leaf wins when set. Sized so a receipt stays fresh across the producer's full period plus slack composeMemoryCoreHealthcheckfoldHeavyMaintenanceStarvation replaces the borrowed deploymentStateBridge.staleAfterMs at toolService.mjs:361; shipped with its consumer
orchestrator.deploymentStateBridge.staleAfterMs existing leaf unchanged remains the bridge-write staleness bound for snapshot-level freshness deployment-state bridge readers this ticket only stops the starvation fold from borrowing it; other consumers keep it
pair reachability assertion boot-time validator add refuses a consumer window shorter than the producer cadence at orchestrator start, mirroring describeMemoryWindowReachability's contract and failure text bootOrchestratorCli / orchestrator start fails closed before state dir, PID file, or log are written
unknown posture mapping fold behaviour fix a fresh posture: unknown maps to a state that neither degrades nor asserts green; the all-clear line is withheld MCP healthcheck, docker healthcheck, container-health controllers explicitly NOT consumed-clear

Decision Record impact

aligned-with ADR 0019 — the formula shape is §10.5's sanctioned remedy for a child derived from a relocatable parent leaf; no re-derivation, no second resolver, env binding stays with the leaf.

Acceptance Criteria

  • A continuously starved plane reports status: degraded with the all-clear line withdrawn on every poll across a full producer period — not only within a window after each stamp. Witnessed by a spec that advances time across at least two producer cadences.
  • The receipt-freshness bound derives from the watchdog cadence, not from deploymentStateBridge.staleAfterMs; re-tuning the cadence moves the window with it (witnessed both directions).
  • A consumer window shorter than the producer cadence is refused at orchestrator boot, with a named reason, before any state directory, PID file, or log is written.
  • A fresh posture: unknown neither degrades health nor asserts green, and the all-clear line is withheld — the consumed-clear mapping is gone (red-proofed against the current behaviour).
  • The existing #17049 guarantees are preserved and re-proved: stale/unavailable/disabled never degrade, unhealthy wins, ensureHealthy() admission still never carries the fold.
  • Mutation control: a spec arm that fails if the fix is reverted, for each of the two invariants.

Out of Scope

Changing the watchdog cadence itself (the 10-minute choice is deliberate and documented) · the fairness/admission gate, which owns the starvation condition rather than its reporting · deploymentStateBridge.staleAfterMs for its own snapshot-freshness consumers · any active alarm dispatcher — the consumed aggregate verdict remains the signal.

Avoided Traps

  • Loosening the freshness guard instead of fixing the bound. The !fresh → do not degrade rule is a deliberate safety property from #17049 (inconclusive never authorizes degradation). Relaxing it would trade a false-green for a false-degrade. The guard is right; the number handed to it is wrong.
  • Treating this as a tuning problem. Option (c) makes today's numbers agree without making them stay agreed. The pair must be coupled by construction or asserted at boot — ideally both.
  • Re-deriving the watchdog cadence at the consumer. That would be ADR-0019 A1/A3; the sanctioned shape is a formula over the resolved leaf.

Related

#17049 (closed; the parent whose two invariants these are) · PR #17099 (835b36653, the merge that delivered the rest) · #16561 (the originating starvation work) · ADR-0019 §10.5 · describeMemoryWindowReachability (memoryPressureDisposition.mjs:185) as the in-tree precedent.

Live latest-open sweep: latest 20 open checked 2026-08-17T09:21Z, no equivalent; A2A herd-window sweep (30 messages, all read-states) clean — no [lane-claim]/[lane-intent] on this scope.

Origin Session ID: c992afd0-2e26-410e-b460-b480ccd0a240

Retrieval Hint: query_raw_memories("starvation health fold receipt freshness producer cadence consumer window 10min 2min duty cycle")

tobiu referenced in commit 47f0951 - "feat(ai): refuse a starvation verdict its own consumer cannot read (#17290) (#17292) on Aug 17, 2026, 1:27 PM
tobiu closed this issue on Aug 17, 2026, 1:27 PM