Orchestrator healthcheck treats long tasks as daemon death

Context

The canonical local Orchestrator became Docker-unhealthy for roughly 20 minutes while a legitimate session-summary child was still running. The child completed, orchestrator-state.json advanced, and the same health command passed again. The operational observation is recorded on #16167.

Fresh V-B-A on current source and the live container establishes the mechanism:

• ai/deploy/docker-compose.yml declares health as orchestrator-state.json mtime younger than 10 minutes.
• TaskStateService writes that file on task lifecycle transitions, not on every daemon poll.
• Orchestrator.poll() already pulses the per-role authority lease every poll.
• The live .authority-lease-container-plane continued to carry a fresh lastPulse independently of task completion.

So this is not daemon death and not a deployment-only mismatch. It is a false-negative health contract.

The Problem

Docker health currently answers “did any task cross a lifecycle boundary recently?” A valid child that runs longer than ten minutes makes that answer false even while the Orchestrator event loop remains responsive and continues its ownership heartbeat. Deployment diagnostics then recommend restarting healthy work.

Daemon liveness and task progress are distinct signals. The former must stay green during legitimate long work; the latter must remain visible so a truly wedged task is not hidden.

The Architectural Reality

• ADR 0014 assigns health/readiness semantics to the dedicated Orchestrator container.
• ai/daemons/orchestrator/authorityLease.mjs already owns the per-role liveness contract: .authority-lease-<profile>, refreshed from Orchestrator.poll(), with AUTHORITY_LEASE_TTL_MS.
• TaskStateService owns task lifecycle/progress and must remain that signal.
• ADR 0019 requires the probe to read resolved AiConfig.orchestrator.dataDir and AiConfig.orchestrator.authorityProfile at the use site; it must not re-read environment variables or duplicate config resolution.
• Owning placement is the existing ai/deploy/docker-compose.yml healthcheck, with focused coverage in test/playwright/unit/ai/daemons/orchestrator/daemon.spec.mjs. The mandatory structure-map command was run but failed with Node's maximum-string limit; no new file or placement is proposed.

The Fix

Change the Orchestrator Compose healthcheck to validate the expected role authority lease under the resolved Orchestrator data directory. Reuse authorityLeaseFilename() and AUTHORITY_LEASE_TTL_MS; do not add another heartbeat file or timeout knob.

Keep task-state freshness out of Docker container liveness. Task stalls remain the responsibility of task-state diagnostics and the existing per-lane watchdogs.

Contract Ledger

Decision Record impact

aligned-with ADR 0014 and aligned-with ADR 0019. No new topology, authority class, or config leaf.

Acceptance Criteria

• A fresh expected-role authority lease makes the Compose probe pass even when orchestrator-state.json is older than ten minutes and records a running task.
• A missing, corrupt, wrong-role, or older-than-authority-TTL lease makes the probe fail.
• The probe reads AiConfig.orchestrator.dataDir and AiConfig.orchestrator.authorityProfile at the use site and reuses the authority-lease filename/TTL primitives.
• Focused unit coverage exercises both the long-task false-negative and fail-closed controls.
• docker compose --profile cloud config validates the rendered healthcheck.
• [L3-deferred — post-merge rollout needed] Post-merge rollout evidence distinguishes container liveness from task progress during one legitimate long child.

Out of Scope

• Changing session-summary timeouts, batching, retries, or task-stall policy.
• Changing Docker restart policy.
• Deploying the already-merged #15825 mailbox fix; that remains a cohort rollout concern.
• Reopening either closed predecessor.

Avoided Traps

• Raise the 10-minute threshold: moves the false negative without separating the signals.
• PID-only liveness: cannot prove an event loop is responsive.
• Add a second heartbeat artifact: duplicates the authority lease that already pulses every poll.
• Treat all long tasks as healthy: task-state diagnostics remain separate and may still classify a task as stalled.

Related

Related: #11937

Related: #12025

Related: #15759

Related: #16167

Related: #16223

Duplicate Sweep

Live latest-open sweep: checked the latest 20 open issues on 2026-08-01; no equivalent ticket found. The all-state recent A2A claim sweep found no overlapping claim. Repository issue/discussion search found the #16167 observation and the closed predecessor chain, but no owned fix.

Origin Session ID: 6a945202-a877-4979-be3f-7fca04f97788

Retrieval Hint: query_raw_memories("orchestrator state-file healthcheck false unhealthy long session-summary authority lease") Retrieval Hint: live source anchors ai/deploy/docker-compose.yml, ai/daemons/orchestrator/authorityLease.mjs, and test/playwright/unit/ai/daemons/orchestrator/daemon.spec.mjs.