Cloud-mode orchestrator: spawn chroma ENOENT loop on first boot

Context

Fifth friction signal from the first-real-world cloud-deployment exercise (siblings: #12014 #12015 #12016 #12017). With the cloud profile booted via docker compose --profile cloud up, the orchestrator container reaches unhealthy because its ProcessSupervisor enters an infinite loop trying to launch chroma as a host child process — but in the cloud topology, chroma is a peer container with its own lifecycle, not an orchestrator-managed subprocess.

The other cloud-safe lane gates (primary-dev-sync, kbSync, bridgeDaemon, golden-path-repo-enrichment, MLX) all have explicit env toggles. The chroma-supervision lane does not, so it cannot be turned off declaratively, and the orchestrator log fills with spawn chroma ENOENT every 15 seconds.

The Problem

Boot a stack mirroring ai/deploy/docker-compose.yml --profile cloud --profile local-model (or any cloud-mode deployment); container logs confirm:

[Orchestrator] Started. summaryInterval=600000ms kbSyncInterval=1800000ms poll=3000ms.
[ProcessSupervisor] Starting chroma daemon (supervisor-restart).
[ProcessSupervisor] chroma daemon failed to start: spawn chroma ENOENT
[ProcessSupervisor] Starting chroma daemon (supervisor-restart).
[ProcessSupervisor] chroma daemon failed to start: spawn chroma ENOENT
... repeats every ~15 seconds indefinitely ...

The deploy-image (node:24-alpine-based) has no chroma binary — by design, since the cloud topology delegates Chroma to a separate chroma container reached over the compose network at chroma:8000. The orchestrator process correctly connects to that container for its own data-plane work; only the supervisor restart loop is wrong-shape in cloud mode.

Cloud-mode env vars currently passed (canonical):

- NEO_AI_DEPLOYMENT_MODE=cloud
- NEO_ORCHESTRATOR_PRIMARY_DEV_SYNC_ENABLED=false
- NEO_ORCHESTRATOR_KB_SYNC_ENABLED=false
- NEO_ORCHESTRATOR_BRIDGE_DAEMON_ENABLED=false
- NEO_ORCHESTRATOR_GOLDEN_PATH_REPO_ENRICHMENT_ENABLED=false
- NEO_ORCHESTRATOR_MLX_ENABLED=false

The chroma-supervisor lane is missing from the gate set; NEO_AI_DEPLOYMENT_MODE=cloud is not consulted by the supervisor's lane registry.

Healthcheck consequence: the container is unhealthy after boot. docker compose up --wait returns non-zero on --profile cloud. The deployment otherwise functions (KB, MC, chroma, local-model all healthy; MC add_memory round-trips work once embedding models are pulled), but the orchestrator failure is visible to any monitoring/healthcheck gate.

The Architectural Reality

Affected:

• ai/daemons/orchestrator/Orchestrator.mjs — the lane registry; needs awareness of cloud topology.
• ai/daemons/orchestrator/services/ — ProcessSupervisorService.mjs (introduced by #11022) is the spawning surface.
• ai/deploy/docker-compose.yml:138-144 — orchestrator env block (the gate-toggle list).

Sibling context: ADR 0014 explicitly separates the orchestrator from chroma as distinct cloud-deployable containers; this is the topology the lane registry must match.

The Fix

Two viable shapes (implementer's call):

1. Implicit gate via deployment-mode: in Orchestrator.mjs lane registry, when NEO_AI_DEPLOYMENT_MODE === 'cloud', skip the chroma supervisor lane entirely. Consistent with how primaryDevSync is treated (local-only by design) — same shape, applied to chroma.

2. Explicit toggle: add NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED (default true for backwards compatibility), set =false in ai/deploy/docker-compose.yml cloud env block. Operator-visible; matches the existing toggle pattern for the other five lanes.

Option 1 is the smaller change and matches the spirit of the cloud-vs-local lane split. Option 2 is more discoverable but adds another env var to the contract. No strong preference; implementer should pick whichever fits the existing lane-registry shape better.

Either way: the orchestrator container should become healthy on first cloud-profile boot without the supervisor loop.

Contract Ledger

Decision Record impact

aligned-with ADR 0014 — completes the cloud topology contract for the orchestrator's own supervision boundary.

Acceptance Criteria

• docker compose -f ai/deploy/docker-compose.yml --profile cloud --profile local-model up -d --wait returns 0 against dev HEAD (regression guard).
• Container logs for orchestrator show no spawn chroma ENOENT lines after the first 30 seconds of boot.
• Local-mode behavior unchanged — npm run orchestrator (or the local launcher) still supervises chroma as a child process when NEO_AI_DEPLOYMENT_MODE is unset.
• If Option 2 is chosen: NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED documented alongside the other five orchestrator lane toggles.

Out of Scope

• Reworking the entire ProcessSupervisor service. Scope is the chroma lane gate.
• Cloud-mode behavior of other supervised lanes (currently only chroma is host-vs-peer-ambiguous in the cloud topology).
• Backup-task spawn git ENOENT failure observed in the same logs — separate symptom, separate ticket if filed.

Avoided Traps

• Filing as enhancement — rejected; the cloud-profile deployment is unhealthy on first boot, which is a correctness bug. --profile cloud is currently broken end-to-end at the orchestrator boundary.
• Removing chroma supervision entirely — rejected; local-mode developers still use the single-process orchestrator-supervises-chroma model.

Related

• #11022 — introduced ProcessSupervisorService (the surface that needs the gate).
• #11844 — orchestrator co-location refactor (where the lane registry now lives).
• #11722 — deployment maintenance config (where the cloud-mode env toggles were established).
• ADR 0014 — cloud topology authority (the boundary that chroma-supervision crosses in cloud mode).
• #12014 — sibling first-real-world deployment friction.
• #12015 / #12016 / #12017 — siblings from the same exercise.

Handoff Retrieval Hints

• Container log evidence: docker logs <orchestrator-container> 2>&1 | grep "spawn chroma ENOENT" on a freshly booted cloud-profile stack.
• Semantic: "orchestrator cloud-mode chroma supervisor lane", "ProcessSupervisor spawn ENOENT loop".
• File anchors: ai/daemons/orchestrator/Orchestrator.mjs lane registry, ai/daemons/orchestrator/services/ProcessSupervisorService.mjs, ai/deploy/docker-compose.yml:138-144.