Orchestrator healthcheck probes wrong filename (state.json vs orchestrator-state.json)

Context

Seventh friction signal from the first-real-world cloud-deployment exercise (siblings #12014/#12015/#12016/#12017/#12019/#12022). The cloud-profile orchestrator container --profile cloud boots cleanly and runs maintenance tasks correctly, but never reaches Docker healthy state because the healthcheck filename does not match the file the orchestrator actually writes.

The Problem

ai/deploy/docker-compose.yml:179 defines the healthcheck:

test: ["CMD-SHELL", "node -e \"const fs=require('fs');const f=(process.env.NEO_AI_ORCHESTRATOR_DIR||'/app/.neo-ai-data/orchestrator-daemon')+'/state.json';try{const m=fs.statSync(f).mtimeMs;process.exit(Date.now()-m<600000?0:1)}catch(e){process.exit(1)}\""]

→ probes <NEO_AI_ORCHESTRATOR_DIR>/state.json.

ai/daemons/orchestrator/Orchestrator.mjs:457 initializes the stateFile:

this.stateFile = options.stateFile || path.join(dataDir, 'orchestrator-state.json');

→ writes <NEO_AI_ORCHESTRATOR_DIR>/orchestrator-state.json (prefixed).

TaskStateService.persist() then writes to that path on every task lifecycle transition. Empirical observation on a freshly built cloud-profile orchestrator container:

$ docker exec <deployment->orchestrator ls -la /app/.neo-ai-data/orchestrator-daemon/
total 20
-rw-r--r-- 1 root root    1 May 26 13:26 orchestrator-daemon.pid
-rw-r--r-- 1 root root 1978 May 26 13:28 orchestrator-state.json
-rw-r--r-- 1 root root 1676 May 26 13:28 orchestrator.log

state.json does not exist; the healthcheck try { fs.statSync(f) } throws ENOENT every probe, exits 1, container remains permanently unhealthy. docker compose --profile cloud up -d --wait returns non-zero.

The daemon itself is functional — logs show [Orchestrator] Started, session summarization completed successfully, memory core backup completed successfully, etc. Only the docker-health label is wrong.

V-B-A: changing the mirror compose healthcheck path from /state.json to /orchestrator-state.json (single-line edit) immediately produces a healthy container with no other changes.

The Architectural Reality

Affected:

• ai/deploy/docker-compose.yml:179 — the healthcheck path.
• Counterpart confusion in ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:571 JSDoc, which states "the orchestrator state file (<DEFAULT_DATA_DIR>/state.json per TaskDefinitions.mjs)" — same drift in the documentation comment.
• Source of truth: Orchestrator.mjs:457 + TaskStateService consumer, both reference orchestrator-state.json.

The healthcheck was added by PR #11937 ("docker-compose.yml: add orchestrator + ingress healthchecks for cloud-deployment trial operator visibility"). At that time the canonical filename was either pre-rename, or the healthcheck author didn't run the canonical compose end-to-end against a fresh orchestrator container. Either way, the mismatch shipped.

The Fix

Two viable shapes. Implementer's call:

1. Update the healthcheck path (smallest, safest — single-file change):

      test: ["CMD-SHELL", "node -e \"... +'/orchestrator-state.json'; ...\""]

2. Rename the file in Orchestrator.mjs:457 from orchestrator-state.json to state.json. Requires sweep of any consumers that expect the current name (backup tooling, observability tooling, external operators reading the file directly). Higher blast radius; only do this if there's an architectural reason to prefer the shorter name.

Option 1 is the clean fix. Option 2 is mentioned for completeness.

Also: fix the TenantRepoSyncService.mjs:571 JSDoc to cite the correct filename, regardless of which fix shape lands.

Contract Ledger

Decision Record impact

aligned-with ADR 0014 — implementation alignment under existing cloud-topology authority. No ADR successor risk.

Acceptance Criteria

• ai/deploy/docker-compose.yml orchestrator healthcheck path matches the filename Orchestrator.mjs writes (either by updating the probe path or by renaming the written file).
• docker compose -f ai/deploy/docker-compose.yml --profile cloud up -d --wait returns 0 against dev HEAD (regression guard).
• docker inspect <orchestrator-container> --format '{{.State.Health.Status}}' returns healthy within start_period + interval*retries after first boot.
• TenantRepoSyncService.mjs:571 JSDoc reflects the canonical filename.
• grep -rn "/state\.json\b" ai/ | grep -v node_modules returns zero matches OR each match is explicitly justified (regression guard against re-drift).

Out of Scope

• Other orchestrator-state-related concerns (e.g., spawn git ENOENT from backup task — separate ticket-worthy item).
• Renaming NEO_AI_ORCHESTRATOR_DIR or the directory layout (.neo-ai-data/orchestrator-daemon/).
• Migrating healthcheck format to invoke mcpHealthcheck.mjs against the orchestrator (orchestrator has no MCP endpoint; the mtime check is correct shape).

Avoided Traps

• Filing as documentation only — rejected; the compose healthcheck is functionally broken, not just mis-documented. bug + refactoring captures both surfaces.
• Skipping the docs-fix piece — rejected; leaving JSDoc cite of state.json would re-introduce drift on the next refactor cycle.

Related

• #11937 — originating PR for the healthcheck (closed); shipped the path mismatch.
• #11833, #11844 — Orchestrator refactors; the stateFile default may have evolved during these without the healthcheck path being audited.
• #12014, #12015, #12016, #12017, #12019, #12022 — sibling friction tickets from the same external-deployment exercise.
• ADR 0014 — cloud-deployment topology authority.

Handoff Retrieval Hints

• Empirical: boot --profile cloud, then docker exec <orch> ls /app/.neo-ai-data/orchestrator-daemon/. Shows orchestrator-state.json only; no state.json.
• Source anchor: grep -n 'state\.json' ai/daemons/orchestrator/Orchestrator.mjs ai/deploy/docker-compose.yml ai/daemons/orchestrator/services/*.mjs.
• Semantic: "orchestrator healthcheck state.json filename mismatch", "cloud-profile orchestrator permanently unhealthy".