docker-compose.yml: add orchestrator + ingress healthchecks for cloud-deployment trial operator visibility

Context

40h cloud-deployment trial pending (~2026-05-26). Production ai/deploy/docker-compose.yml has healthchecks for chroma, kb-server, mc-server but not for orchestrator (cloud profile) or ingress (caddy reverse proxy). For a trial where the deployment operator needs real-time visibility into Agent OS state, missing healthchecks on 2 of 5 services creates monitoring gaps.

Distinct from #11791 (tenant-repo ingestion docs + telemetry — focused on the pull-based ingestion lane shipping under #11731). This ticket is about general docker-substrate health visibility across ALL deployed services.

The Problem

orchestrator service in docker-compose.yml lines 132-177:

• Maintenance daemon — no HTTP endpoint to ping
• TaskStateService writes a JSON state file at ${NEO_AI_ORCHESTRATOR_DIR:-.neo-ai-data/orchestrator-daemon}/state.json on every task lifecycle event
• A fresh state file (mtime within last N minutes) = orchestrator is alive + polling

ingress service (caddy) in lines 184-203:

• Has TCP port 443 exposed
• No healthcheck declared — operator can't distinguish "caddy crashed" from "no requests arriving"
• Standard caddy admin endpoint at :2019/config/ (admin API) OR simpler TCP check on 443

The Fix

For orchestrator — add healthcheck verifying state file freshness:

orchestrator:
  # ... existing config ...
  healthcheck:
    test: ["CMD-SHELL", "node -e \"const fs=require('fs');const f=process.env.NEO_AI_ORCHESTRATOR_DIR+'/state.json';const m=fs.statSync(f).mtimeMs;process.exit(Date.now()-m<600000?0:1)\""]
    interval: 60s
    timeout: 5s
    retries: 3
    start_period: 90s

Polls every 60s; state file must be mutated within the last 10 minutes (Orchestrator's swarmHeartbeatIntervalMs + safety margin). 90s start period accommodates first-poll latency.

For ingress — add TCP-port healthcheck on 443:

ingress:
  # ... existing config ...
  healthcheck:
    test: ["CMD-SHELL", "wget --no-check-certificate --quiet --tries=1 --spider https://127.0.0.1/ 2>&1 | grep -E '^(HTTP|Connecting)' || exit 1"]
    interval: 30s
    timeout: 5s
    retries: 3
    start_period: 30s

(Alternative: bind-mount a tiny caddy-admin probe; rejected as scope-creep for trial-window.)

Acceptance Criteria

• orchestrator service in docker-compose.yml has a healthcheck block verifying state file mtime within tolerance window — PR #11939
• ingress service has a healthcheck block verifying caddy port 443 responsiveness — PR #11939 (final shape: nc -z 127.0.0.1 443, TCP-level, per cycle-1 RA on silent-success-empty-output failure mode)
• docker compose config validates clean (no schema errors) — deferred to operator pre-trial validation (PR review host has no local docker; YAML-library structure parse confirms all 6 services have valid healthcheck blocks as preflight evidence)
• Local manual smoke test: docker compose --profile cloud --profile ingress up; observe docker ps shows both services in (healthy) state after start_period — deferred to operator pre-trial validation (same reason)

Deferred Evidence (PR #11939)

PR #11939 satisfied AC1 + AC2 mechanically (compose-file edits + YAML structure validation via yaml library). AC3 + AC4 require live docker not available on the review host; both are formally deferred to operator pre-trial smoke test per the #11939 FAIR-band/Evidence framing. Residual operator actions:

1. Run docker compose config (parses + validates compose-file against docker schema)
2. Run docker compose --profile cloud --profile ingress up and observe docker ps reports (healthy) for both orchestrator (90s start_period) and ingress (30s start_period)
3. If either fails, healthcheck command tuning is small follow-up PR shape; no service-config change needed

This deferred-evidence annotation matches PR #11939's AC3-partial / AC4-deferred state so the close-target issue and the PR substrate stay aligned (per pull-request §10 Evidence Audit).

Out of Scope

• Tenant-repo ingestion health/telemetry → owned by #11791
• Active push-style alerting / external observability integration (e.g., Prometheus/Grafana export) → post-MVP
• chroma / kb-server / mc-server healthcheck tuning (already declared)

Avoided Traps

• Trap: HTTP healthcheck on the orchestrator (it doesn't expose one). Rejection: state-file-mtime is the canonical liveness signal already present in the substrate; introducing an HTTP server would balloon scope.
• Trap: caddy admin API healthcheck on :2019. Rejection: admin port is internal-only by default; activating it for healthcheck would expose extra surface area for marginal value beyond TCP-443 check.
• Trap: tight interval: 10s matching the other services. Rejection: orchestrator polling cadence is minutes, not seconds; 60s interval matches the actual lane semantics + reduces healthcheck noise.

Empirical Anchor

Cloud-deployment trial readiness review 2026-05-24. Operator's framing: "in theory we are there → many new deployment guides, orchestrator enhancements, docker, integration tests." This ticket addresses the docker substrate slice — orchestrator + ingress healthcheck gap was visible on inspection of docker-compose.yml.

— @neo-opus-ada