Orchestrator must guarantee a single Chroma daemon (probe + host-pin)

Context

While tracking down #12133 (KB ask_knowledge_base / query_documents throwing Error executing plan: Internal error: Error finding id on any non-'all' type), the root cause turned out NOT to be the QueryService where-filter — the #12135 fallback only masked the symptom. It was two ChromaDB daemons running concurrently over the one unified persist dir .neo-ai-data/chroma/knowledge-base (which, post KB+MC unification onto a single chroma daemon, holds BOTH the KB neo-knowledge-base collection AND Memory Core's neo-agent-memory / neo-agent-sessions).

Verified 2026-05-28 via ps / lsof:

• PID 9078 — chroma run --path .neo-ai-data/chroma/knowledge-base --port 8000 (orchestrator-spawned; no --host → bound IPv6 [::1]:8000).
• PID 8435 — chroma run --path ./.neo-ai-data/chroma/knowledge-base --host 127.0.0.1 --port 8000 (untracked/external; bound IPv4 127.0.0.1:8000).

A read-only differential probe against each daemon returned the identical Error finding id and listed identical collections — two server processes flushing the same on-disk HNSW/SQLite segment → persistent corruption. The duplicate (8435) was killed this session, reducing to one daemon and stopping further corruption.

The Problem

The orchestrator's single-daemon guarantee rests entirely on its own PID-file bookkeeping; it never checks whether the port is actually already served. Any chroma started outside the orchestrator's tracking — manual npm run ai:server, the manage_database MCP tool, a pre-unification leftover, or a second orchestrator — is invisible, so the supervisor spawns a duplicate. Because the daemon binds dual-stack (no --host) and the client host is localhost (resolves to both families), the two processes silently coexist on :8000 split across IPv4/IPv6.

The Architectural Reality

• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:319 runTask — guards only on state.running (set by recoverTask adopting its own chroma.pid). No port/heartbeat probe before spawnFn.
• ai/daemons/orchestrator/Orchestrator.mjs:618 poll() — continuousTasks includes chroma; when !state.running and the restart cooldown elapsed, calls runTask('chroma', 'supervisor-restart').
• ai/daemons/orchestrator/TaskDefinitions.mjs:57 — chroma args ['run','--path','.neo-ai-data/chroma/knowledge-base','--port','8000'], no --host → dual-stack bind.
• ai/config.template.mjs engines.chroma.host: 'localhost' → resolves to both 127.0.0.1 and ::1; chromadb clients route nondeterministically.
• package.json ai:server — chroma run --path ./.neo-ai-data/chroma/knowledge-base (no --port / --host); an out-of-band spawn path with no arbitration.
• Reusable readiness primitive already exists: ai/services/shared/vector/chromaClientPrimitives.mjs:78 chromaConnect({client, logger}) (non-throwing heartbeat).

The Fix

1. Pre-spawn readiness probe in ProcessSupervisorService.runTask: for port-owning tasks, before spawnFn, probe whether the endpoint is already serving. Add an optional alreadyRunningProbe to the task definition (symmetric to the existing postSpawn hook); the chroma task supplies a chromaConnect-based probe against engines.chroma.{host,port}. If already serving → adopt/skip (record outcome) instead of spawning. Closes the "untracked daemon invisible to state.running" gap.
2. Pin a single address family: add --host 127.0.0.1 to the chroma task-def and the ai:server script; default engines.chroma.host to 127.0.0.1 (not localhost). A second chroma run then fails loudly with EADDRINUSE instead of silently binding the other family, and client routing becomes deterministic.
   • Rollout note: applying the host-pin requires restarting the chroma daemon (the live one is currently IPv6-only). Post-merge operator step.

Contract Ledger

Decision Record impact

none (no ADR challenged/superseded). Establishes the orchestrator as the single owner of chroma daemon lifecycle — see the external-only follow-up under Related.

Acceptance Criteria

• ProcessSupervisorService.runTask does not spawn a port-owning task whose endpoint is already serving (probe-gated). Unit test: mock spawnFn + a probe returning "already up" → asserts no spawn and a recorded skipped/adopt outcome.
• chroma task-def and ai:server pin --host 127.0.0.1; engines.chroma.host defaults to 127.0.0.1.
• Unit test: probe returning "not up" still spawns (no regression to existing supervise/restart behavior).
• Non-port tasks are unaffected (probe is opt-in per task-def).
• (Post-merge, operator) restart the chroma daemon; confirm a single IPv4-bound daemon on :8000 and that a second ai:server fails with EADDRINUSE.

Out of Scope

• The broader external-only chroma mode cleanup (remove manage_database start/stop, autoStartDatabase, the per-subsystem lifecycle services, simplify healthchecks, collapse/simplify the two ChromaManagers, orchestrator as single source of truth) — separate follow-up so this safety fix stays small.
• The one-time rebuild of the already-corrupted neo-knowledge-base collection (operator op; NOT npm run ai:defrag-kb, whose orphan-cleanup ai/scripts/maintenance/defragChromaDB.mjs:471 would delete MC's segment dirs in the unified dir → data loss).
• Fixing defragChromaDB.mjs's unified-store unsafety (separate ticket).

Related

• #12133 — symptom (KB typed-query Error finding id); this is its root cause.
• #12135 — rejected query-time fallback.
• Follow-up (to be filed): external-only chroma mode / orchestrator-as-single-source-of-truth cleanup.

Origin Session ID: 51ac9efd-4b5a-488e-b3dd-aa2673acbec9 Retrieval Hint: "two chroma daemons IPv4 IPv6 unified store Error finding id orchestrator single daemon probe host-pin"

Filed by @neo-opus-ada.