Orchestrator recycles Chroma after a max runtime (kill→restart→defrag)

Context

PR #12137 makes the orchestrator reap duplicate Chroma daemons (singularity). Separately, a long-lived Chroma process accumulates two distinct problems: process-level cruft (memory/fd) and on-disk HNSW index fragmentation (we observed 307 segment dirs in the unified store this session). The operator does not want a Chroma daemon running unrecycled for long stretches (e.g. a month).

The Problem

A never-recycled Chroma daemon (a) carries process-level cruft and (b) serves an increasingly fragmented on-disk index. SIGTERM does not work on chroma — empirically confirmed across hundreds of attempts — only kill -9 reliably stops it.

The Architectural Reality

• The orchestrator already supervises chroma as a continuous task (ai/daemons/orchestrator/Orchestrator.mjs poll() + TaskDefinitions.mjs chroma task). Daemon age is trackable via taskState('chroma').lastRunAt (set at spawn; survives orchestrator restarts via persisted state).
• The recycle order MUST be kill → restart → defrag: defragChromaDB.mjs extracts/rebuilds via the ChromaClient API and needs a LIVE daemon, so it runs against the fresh restarted process. kill+restart first heals a corrupted/hung process; defrag then compacts the index on the healthy daemon.
• engines.chroma.{host,port} are the unified-store coordinates; Config.maintenance.defrag.intervalMs (7d) already exists as a maintenance-cadence precedent.

The Fix

1. New config orchestrator.chroma.maxRuntimeMs (+ env NEO_CHROMA_MAX_RUNTIME_MS; concrete default in the template, e.g. 24h).
2. Orchestrator: when now - taskState('chroma').lastRunAt > maxRuntimeMs, run the chain — kill -9 the tracked daemon → existing supervisor respawn → once the fresh daemon is chromaConnect-ready, run the safe defrag.
3. Prerequisite (dependency): a unified-store-safe defrag. Today's defragChromaDB.mjs is unsafe for the unified store — its step-6 orphan-cleanup (:471) deletes MC's segment dirs in the shared persist dir → MC data loss. The safe defrag must rebuild ALL collections (KB + MC) and never delete sibling-collection dirs; ideally shadow-swap (rebuild into a shadow collection, atomic-swap) for zero/minimal downtime — KB already has shadow-swap machinery; MC would need it for the unified rebuild.
4. Gated by chromaDaemonEnabled (off in cloud, where chroma is externally managed).

Contract Ledger

Decision Record impact

aligned-with ADR 0003 (unified Chroma topology).

Acceptance Criteria

• orchestrator.chroma.maxRuntimeMs config + env binding; concrete default in the template.
• Orchestrator recycles chroma when daemon age > maxRuntimeMs, in kill→restart→defrag order, gated by chromaDaemonEnabled.
• Recycle uses kill -9 (SIGTERM is not attempted — chroma ignores it).
• Defrag runs only after the restarted daemon is connection-ready, and uses the unified-store-safe defrag.
• Unit tests: max-runtime trigger logic; OS-level kill is a no-op under UNIT_TEST_MODE (no real process touched).
• (post-merge) operator-observed: a daemon older than maxRuntimeMs is recycled and the store is compacted afterward.

Out of Scope

• The kill-dups singularity net (PR #12137).
• External-only mode / removing manage_database etc. (separate ticket).

Related

• #12136 / PR #12137 (kill-dups singularity — this builds on it).
• Sibling: "KB + MC external-only Chroma; orchestrator as single source of truth".

Origin Session ID: 51ac9efd-4b5a-488e-b3dd-aa2673acbec9

Filed by @neo-opus-ada.