Sub of #13624 — hung-lease watchdog INTEGRATION (the orchestrator lease-monitor)
#13761 shipped the pure isHungLeaseHolder decision. This wires it into the live orchestrator: a periodic lease-monitor that ps-samples the active heavy-lease holder's cpu and force-releases + records a failed outcome when the holder is HUNG (alive + within-TTL but sustained-idle — the 06-21 01:17 incident, invisible to the pid/TTL stale-check until the freeze).
Design (decide-and-document)
- A
createLeaseMonitor factory with a tick() method (seam-injectable: inspectLease, sampleCpuPercent, releaseLease, recordOutcome) — pure-testable via the seams.
- Per sample-interval (SLOW — minutes-scale, per Grace's #13761 forward-flag: a tight
minConsecutiveIdle × 250ms-tick = 750ms would false-positive a <1s I/O-blocked holder, re-creating the DRAIN): inspect the active lease → ps-sample the holder pid's cpu% → push to a capped per-pid history → isHungLeaseHolder(history) → if hung: releaseLease + recordTaskOutcome(owner, 'failed', {reason:'watchdog-released-hung-holder', failurePhase:'hung-lease-holder'}).
- Typed-outcome state =
'failed', not 'skipped' (per #13765 boundary-5 / @neo-gpt): a hung holder RAN (held the lease) then stalled — that is a FAILURE, distinct from skipped (=never-ran) / deferred (=postponed) / task-state markSkipped. Uses the EXISTING recordTaskOutcome 'failed' state; the convergence may add a hung-specific typed-outcome later.
- Fail-SAFE: a bad/non-finite cpu sample RESETS the idle window (never bridges a
0,0,0,NaN,0 gap into a false consecutive-idle run); a release/record seam exception returns release-failed without breaking the orchestrator loop.
Out of scope
The daemon-loop hook (the 1-line monitor.tick() wiring) lands LAST, after the service is tested — minimizing the live-daemon blast. Interop contract (@neo-gpt #13769): on a force-release the hook composes BOTH the health failed (recordOutcome) AND the task-state markSkipped (TaskStateService), unless the owner has no task-state row by design.
Acceptance Criteria
Sub of #13624. Builds on #13761 (imports isHungLeaseHolder).
Sub of #13624 — hung-lease watchdog INTEGRATION (the orchestrator lease-monitor)
#13761 shipped the pure
isHungLeaseHolderdecision. This wires it into the live orchestrator: a periodic lease-monitor that ps-samples the active heavy-lease holder's cpu and force-releases + records afailedoutcome when the holder is HUNG (alive + within-TTL but sustained-idle — the 06-21 01:17 incident, invisible to the pid/TTL stale-check until the freeze).Design (decide-and-document)
createLeaseMonitorfactory with atick()method (seam-injectable:inspectLease,sampleCpuPercent,releaseLease,recordOutcome) — pure-testable via the seams.minConsecutiveIdle × 250ms-tick= 750ms would false-positive a <1s I/O-blocked holder, re-creating the DRAIN): inspect the active lease → ps-sample the holder pid's cpu% → push to a capped per-pid history →isHungLeaseHolder(history)→ if hung:releaseLease+recordTaskOutcome(owner, 'failed', {reason:'watchdog-released-hung-holder', failurePhase:'hung-lease-holder'}).'failed', not'skipped'(per #13765 boundary-5 / @neo-gpt): a hung holder RAN (held the lease) then stalled — that is a FAILURE, distinct fromskipped(=never-ran) /deferred(=postponed) / task-statemarkSkipped. Uses the EXISTINGrecordTaskOutcome'failed'state; the convergence may add a hung-specific typed-outcome later.0,0,0,NaN,0gap into a false consecutive-idle run); a release/record seam exception returnsrelease-failedwithout breaking the orchestrator loop.Out of scope
The daemon-loop hook (the 1-line
monitor.tick()wiring) lands LAST, after the service is tested — minimizing the live-daemon blast. Interop contract (@neo-gpt #13769): on a force-release the hook composes BOTH the healthfailed(recordOutcome) AND the task-statemarkSkipped(TaskStateService), unless the owner has no task-state row by design.Acceptance Criteria
createLeaseMonitor(testable seams); per-pid capped cpu-history; force-release +recordTaskOutcome('failed')on a sustained-idle hung holder.minConsecutiveIdle × sample-interval= minutes (not 750ms) — Grace's interval-coupling guard.failed; slow-but-progressing → NOT released;0,0,0,NaN,0→ NOT released; no-active-lease → no-op; ps/inspect-failure + seam-exception → fail-safe.Sub of #13624. Builds on #13761 (imports
isHungLeaseHolder).