Chroma PID adoption accepts process-alive but service-dead

Context

Operator report on 2026-06-28: after pulling dev and restarting the local orchestrator, the log showed the supervisor adopting an existing Chroma process that was still present at the OS process layer but functionally dead. The orchestrator then froze until Chroma was manually killed. This happened immediately after the recent v13.1 self-healing/orchestrator fixes, including #14291/#14292 (singleton-port defer adoption) and #14293/#14294 (Chroma recycle coordination with kbSync).

Fresh V-B-A from this session:

• git log --oneline -8 shows #14294 and #14292 as the last relevant merged PRs before the report.
• ps -axo pid,ppid,stat,command after the manual kill/restart showed one live orchestrator and one Chroma listener, so the current process table is no longer wedged.
• lsof -nP -iTCP:8000 -sTCP:LISTEN showed Chroma listening on IPv6 [::1]:8000.
• Fetch probes verified the live daemon responds on localhost / [::1] for /api/v2/heartbeat and /api/v2/healthcheck, while 127.0.0.1 fails for the current bind.
• Code inspection shows Chroma has singletonPort and default SIGKILL duplicate reaping, but no livenessProbe or healthProbe in ai/daemons/orchestrator/taskDefinitions.mjs.
• Code inspection shows ProcessSupervisorService.recoverTask() adopts a PID-file match using only process.kill(pid, 0) plus processCommand(pid).includes(expectedCommand); it does not prove the Chroma HTTP service is usable before setting running=true.

Live latest-open sweep: checked latest 20 open issues on 2026-06-28 before filing; no equivalent narrow ticket found. Closest open parent is #14039 (Agent OS Stability & Self-Healing). Recent A2A sweep (list_messages({status:'all', limit:30})) showed this is a follow-up to #14293/#14294, not a competing peer claim.

The Problem

The Chroma single-daemon machinery currently distinguishes duplicate processes, but not process-alive/service-dead adoption.

ProcessSupervisorService.recoverTask() treats a persisted chroma.pid as healthy when the PID exists and the command line contains chroma. For Chroma, that is insufficient: a hung or half-dead Chroma process can still satisfy process.kill(pid, 0) and match the command, while its HTTP API is unavailable or wedged. Once adopted, TaskStateService marks the task running. ProcessSupervisorService.superviseTask() then returns early for running tasks unless the task defines a healthProbe. Chroma defines none, so the supervisor never recycles the service-dead process.

This is exactly the worst failure mode for self-healing: the orchestrator believes it healed/adopted the daemon, but actually preserves the bad process and blocks the system until a human kills it.

The Architectural Reality

• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:135-156 - PID-file recovery adopts any PID whose command includes expectedCommand; no readiness or health proof.
• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:1120-1132 - killProcess() intentionally uses SIGKILL; this must stay true for Chroma because SIGTERM is unreliable.
• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:1135-1149 - killTask() recycles a tracked task by SIGKILL + markRecycled; this is the right actuator for an adopted-but-dead Chroma process.
• ai/daemons/orchestrator/taskDefinitions.mjs:153-165 - the Chroma task has command, args, pidFileName, expectedCommand, and singletonPort, but no livenessProbe or healthProbe.
• Closed #12136 originally considered probe/host-pin for Chroma singularity, but the delivered fix pivoted to active duplicate reaping. That closed the duplicate-writer class, not the adopted service-dead class.
• Closed #12138 added max-runtime Chroma recycle and preserved the invariant that Chroma recycle uses SIGKILL.
• Open #14291 / merged #14292 added defer-policy adoption for externally owned port holders. Chroma must not become defer; it is orchestrator-owned and should keep authoritative SIGKILL reaping.

The Fix

Add Chroma service-health proof to the existing supervisor contract without reviving host-pin or defer-policy semantics.

Concrete target shape:

1. Add a Chroma HTTP readiness/health helper in the orchestrator task-definition layer, using the v2 Chroma endpoint that the live process serves (/api/v2/heartbeat or /api/v2/healthcheck) and host semantics that work for the current daemon (localhost/configured host, not hardcoded 127.0.0.1).
2. Wire the Chroma task with a healthProbe so an adopted/running-but-unhealthy Chroma process is recycled by the existing gateRecycleOnHealthProbe() -> killTask('chroma', 'supervisor-health-recycle') path.
3. Consider adding a livenessProbe for the not-running path only if it does not weaken the authoritative Chroma ownership model. A live matching Chroma listener with no tracked canonical PID should still be reconciled by the default policy, not silently treated like shared external infra.
4. Add unit coverage for an adopted Chroma state (running=true, pid set) whose health probe returns false: the supervisor must call killTask, which uses SIGKILL via killProcess, marks recycled, and allows the next poll to spawn fresh Chroma.

Decision Record impact

none - this is a behavior fix inside the existing orchestrator-owned Chroma lifecycle. It preserves the decisions from #12136 (authoritative duplicate reaping) and #12138 (SIGKILL recycle), and does not amend ADR authority.

Acceptance Criteria

• Chroma task definition includes a service-health probe that verifies the Chroma HTTP API, not merely PID existence or TCP bind.
• A recovered/adopted Chroma process whose health probe reports unhealthy is recycled through the existing killTask('chroma', ...) path; the stale running=true state must not persist indefinitely.
• Chroma recycle continues to use SIGKILL; no SIGTERM-only shutdown path is introduced.
• Default singleton-port reaping for Chroma remains authoritative; Chroma is not converted to duplicateListenerPolicy: 'defer'.
• Unit coverage in ProcessSupervisorService.spec.mjs and/or Orchestrator.spec.mjs covers running/adopted Chroma health failure -> recycle -> next-poll restart eligibility.
• Manual/local validation documents the expected endpoint behavior for the current daemon bind (localhost/[::1] succeeds; 127.0.0.1 may fail when Chroma binds IPv6-only).

Out of Scope

• Reverting #14292 defer-policy adoption for dev-server/Neural Link Bridge.
• Reverting #14294 heavy-maintenance lease coordination.
• Changing Chroma from orchestrator-owned to externally owned/shared-infra semantics.
• Re-architecting the full Chroma recycle/defrag cadence.

Avoided Traps

• Switching Chroma to graceful SIGTERM - rejected. Closed #12138 and live operator evidence both say Chroma does not shut down reliably via SIGTERM; SIGKILL is the safety primitive.
• Making Chroma defer because defer adoption was just added - rejected. defer is for shared external infra; Chroma writes the unified persist dir and must remain authoritative.
• Only checking TCP connect - insufficient. A listener can exist while the Chroma service is wedged. Prefer a Chroma HTTP heartbeat/healthcheck response.
• Reviving the old host-pin as the primary fix - rejected. Closed #12136 pivoted away from host-pin; this ticket is about service-health after PID adoption.

Related

• Parent epic: #14039 - Agent OS Stability & Self-Healing (v13.1).
• #12136 - Chroma single-daemon guarantee; duplicate reaping delivered, service-dead adoption gap remains.
• #12138 - Chroma max-runtime recycle; SIGKILL invariant.
• #14291/#14292 - defer-policy port adoption for shared infra; not the Chroma ownership model.
• #14293/#14294 - Chroma recycle coordination with kbSync; this is the next Chroma lifecycle regression.

Origin Session ID: 019f0f79-7af4-7a70-893a-dd58e03337d6

Handoff Retrieval Hint: "Chroma PID adoption process-alive service-dead healthProbe ProcessSupervisor recoverTask SIGKILL recycle"