LearnNewsExamplesServices
Frontmatter
id13766
titlefeat(ai): hung-lease watchdog INTEGRATION — orchestrator lease-monitor (sub of #13624)
stateClosed
labels
aiarchitecturemodel-experience
assigneesneo-opus-ada
createdAtJun 21, 2026, 2:39 PM
updatedAtJun 21, 2026, 4:00 PM
githubUrlhttps://github.com/neomjs/neo/issues/13766
authorneo-opus-ada
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 21, 2026, 4:00 PM

feat(ai): hung-lease watchdog INTEGRATION — orchestrator lease-monitor (sub of #13624)

Closed v13.1.0/archive-v13-1-0-chunk-5 aiarchitecturemodel-experience
neo-opus-ada
neo-opus-ada commented on Jun 21, 2026, 2:39 PM

Sub of #13624 — hung-lease watchdog INTEGRATION (the orchestrator lease-monitor)

#13761 shipped the pure isHungLeaseHolder decision. This wires it into the live orchestrator: a periodic lease-monitor that ps-samples the active heavy-lease holder's cpu and force-releases + records a failed outcome when the holder is HUNG (alive + within-TTL but sustained-idle — the 06-21 01:17 incident, invisible to the pid/TTL stale-check until the freeze).

Design (decide-and-document)

  • A createLeaseMonitor factory with a tick() method (seam-injectable: inspectLease, sampleCpuPercent, releaseLease, recordOutcome) — pure-testable via the seams.
  • Per sample-interval (SLOW — minutes-scale, per Grace's #13761 forward-flag: a tight minConsecutiveIdle × 250ms-tick = 750ms would false-positive a <1s I/O-blocked holder, re-creating the DRAIN): inspect the active lease → ps-sample the holder pid's cpu% → push to a capped per-pid history → isHungLeaseHolder(history) → if hung: releaseLease + recordTaskOutcome(owner, 'failed', {reason:'watchdog-released-hung-holder', failurePhase:'hung-lease-holder'}).
  • Typed-outcome state = 'failed', not 'skipped' (per #13765 boundary-5 / @neo-gpt): a hung holder RAN (held the lease) then stalled — that is a FAILURE, distinct from skipped (=never-ran) / deferred (=postponed) / task-state markSkipped. Uses the EXISTING recordTaskOutcome 'failed' state; the convergence may add a hung-specific typed-outcome later.
  • Fail-SAFE: a bad/non-finite cpu sample RESETS the idle window (never bridges a 0,0,0,NaN,0 gap into a false consecutive-idle run); a release/record seam exception returns release-failed without breaking the orchestrator loop.

Out of scope

The daemon-loop hook (the 1-line monitor.tick() wiring) lands LAST, after the service is tested — minimizing the live-daemon blast. Interop contract (@neo-gpt #13769): on a force-release the hook composes BOTH the health failed (recordOutcome) AND the task-state markSkipped (TaskStateService), unless the owner has no task-state row by design.

Acceptance Criteria

  • createLeaseMonitor (testable seams); per-pid capped cpu-history; force-release + recordTaskOutcome('failed') on a sustained-idle hung holder.
  • minConsecutiveIdle × sample-interval = minutes (not 750ms) — Grace's interval-coupling guard.
  • Fail-SAFE: a bad sample resets the window; a release/record seam exception never throws.
  • Tests: hung holder → released + outcome failed; slow-but-progressing → NOT released; 0,0,0,NaN,0 → NOT released; no-active-lease → no-op; ps/inspect-failure + seam-exception → fail-safe.

Sub of #13624. Builds on #13761 (imports isHungLeaseHolder).

tobiu referenced in commit deb90d3 - "feat(ai): hung-lease watchdog integration — orchestrator lease-monitor (sub of #13624) (#13768) on Jun 21, 2026, 4:00 PM
tobiu closed this issue on Jun 21, 2026, 4:00 PM