LearnNewsExamplesServices
Frontmatter
id17049
titleA starving heavy-maintenance waiter must degrade the orchestrator health surface
stateClosed
labels
enhancementaiagent-os
assigneesneo-opus-vega
createdAtAug 13, 2026, 1:57 PM
updatedAtAug 17, 2026, 11:23 AM
githubUrlhttps://github.com/neomjs/neo/issues/17049
authorneo-opus-vega
commentsCount4
parentIssue17072
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 17, 2026, 11:20 AM

A starving heavy-maintenance waiter must degrade the orchestrator health surface

Closed Backlog/active-chunk-15 enhancementaiagent-os
neo-opus-vega
neo-opus-vega commented on Aug 13, 2026, 1:57 PM

Context

Successor slice of #16561, split out so the delivered halves stay truthful: the measurement half (durable deferralStreakStartedAt, merged) and the fairness/admission half (durable waiter ledger + three-rank yield gate + wrapped CLI) close the starvation itself. What remains is #16561's fix-shape item 2: reporting starvation as a health fact.

Today a plane can defer priority work for hours while the orchestrator health surface reads green — each deferral is one log line, and no aggregate exists. The waiter ledger (ai/daemons/orchestrator/services/heavyMaintenanceWaiterLedger.mjs) now makes the aggregate cheap to compute: every deferred task maintains one durable entry carrying deferredSince, priorityZero, bootstrapCritical.

AC

(Amended 2026-08-14 per the PR's Cycle-1 review — the original "fail-open to green" contract was replaced by the four-posture contract during implementation without amending this body; this amendment truth-folds the ticket to the delivered contract, and the original AC list is preserved in the correction comment.)

  • The starvation fact is a consumed health verdict, not observable bytes: any live waiter whose streak exceeds the risk bound degrades aggregate health (the Memory Core healthcheck's top-level status, with "All features are operational" withdrawn), carrying a receipt naming the waiter, its class, deferredSince, and the current lease holder.
  • The risk bound is an AiConfig leaf (heavyMaintenanceLease.starvationDegradeAfterMs) introduced in the same PR as its consumer.
  • Waiter freshness reads the ADMISSION authority (WAITER_ENTRY_STALE_AFTER_MS, 10 min) — never the 6h lease-holder TTL — so health expires a dead waiter on the same clock the fairness gate does; an entry expired for admission clears health on the next check.
  • The verdict is four-posture, recomputed per check, never latched: degraded (readable breach — readable evidence beats unreadable noise) · healthy · unknown (unreadable entries with no readable breach, or a watchdog fault — neither degrades nor asserts green; inconclusive never authorizes degradation) · disabled (bound off).
  • The monitor is not starvable by the condition it observes: health-check lanes dispatch alongside the single per-poll winner, so a perpetually-due priority-zero task held behind an out-of-process lease cannot silence the watchdog. Heavy-task admission semantics are unchanged.
  • The consuming fold only ever moves healthydegraded: an existing unhealthy verdict wins; a stale receipt, or a stale/schema-degraded/unavailable snapshot, cannot authorize degradation.
  • Spec arms cover: the production composition over a REAL ledger through the REAL collaborator APIs (healthy → degraded → expiry-cleared, verdict terminally persisted); the alongside-dispatch falsifier; the full consumption matrix (degraded/healthy/unknown/disabled × fresh/stale × snapshot states × unhealthy precedence); and the four-posture evaluator including corrupt-ledger unknown.

Contract Ledger

Surface Kind Change Contract Consumers Notes
heavyMaintenanceLease.starvationDegradeAfterMs AiConfig leaf add number, default 1h; <= 0 disables (posture disabled). Sized above fairnessYieldAfterMs BY DEFAULT — the relationship is advisory, not enforced; inversion is noisy, never unsafe starvation watchdog runner shipped with its consumer
intervals.heavyMaintenanceStarvationWatchdogCheckMs AiConfig leaf add number, default 10 min; <= 0 disables the lane scheduling registry cadence of the read-only check
waiter freshness authority reuse consume WAITER_ENTRY_STALE_AFTER_MS (10 min) — the admission clock; the 6h lease TTL is NEVER a waiter-freshness bound watchdog runner via listActiveWaitersSync one clock for admission and health
lease-path resolution collaborator API consume MaintenanceBackpressureService.resolveHeavyMaintenanceLeasePath() — the canonical production method watchdog runner proven by a production-composition spec crossing the real API
four-posture verdict durable task-state field (starvation) add {posture: degraded|healthy|unknown|disabled, checkedAt, degradeAfterMs, waiterCount, unreadableCount, leaseHolder, breaches[]}; recomputed per check, persisted via the terminal task-state write; the catch path persists unknown deployment-state bridge survives restarts; no latch to clear
heavyMaintenanceStarvation deployment-state snapshot section add registered in CURRENT_SNAPSHOT_SECTIONS + ADDITIVE_SNAPSHOT_SECTIONS (tolerated-absent additive; producer metadata declares it); projected verbatim from the persisted verdict by a detached bridge collector inspect_deployment, get_deployment_state_snapshot, MC health fold absent until the first verdict
aggregate-health fold composed MCP healthcheck surface (composeMemoryCoreHealthcheck) add foldHeavyMaintenanceStarvation applied at the COMPOSED surface, riding the composer's request-fresh deployment inspection: only an available snapshot + FRESH receipt (checkedAt within deploymentStateBridge.staleAfterMs) + posture === 'degraded' degrades status; healthy/unknown/disabled/stale/unavailable never degrade; unhealthy wins; the all-clear detail is withdrawn; the consumed observation is recorded as heavyMaintenanceStarvation.state. Admission exclusion is contractual: HealthService.healthcheck() — the ensureHealthy() tool-admission source — never carries the fold, so a starved maintenance lane can never block capabilities it does not affect (semantic recall stays dispatchable) MCP healthcheck tool, docker healthcheck, container-health controllers — and explicitly NOT ensureHealthy() admission recovery is latch-free by per-request composition
health-check dispatch scheduling pipeline change due health-check lanes dispatch alongside the single per-poll winner (lease-free, read-only, self-cadenced); winner selection and heavy admission unchanged all four health-check lanes the monitors are collectively non-starvable

Non-Goals

  • No scheduling/admission changes for heavy tasks — the fairness gate owns those.
  • No lowering of the lease staleAfterMs (unchanged rationale from #16561).
  • No active alarm dispatcher — the consumed aggregate verdict is the signal; sibling watchdogs own their own alarm channels.
tobiu referenced in commit 835b366 - "feat(ai): degrade orchestrator health when a heavy-maintenance waiter starves (#17049) (#17099) on Aug 14, 2026, 2:08 PM
tobiu closed this issue on Aug 14, 2026, 2:08 PM
tobiu
tobiu Aug 17, 2026, 11:20 AM

closed, since the ticket has a merged PR. follow-up tickets are possible.

tobiu closed this issue on Aug 17, 2026, 11:20 AM