Chroma recycle preempts an in-flight kbSync → ChromaConnectionError

Context

The local orchestrator's knowledge-base sync aborted mid-run:

[ProcessSupervisor] Starting knowledge base embedding...
[ProcessSupervisor] Loaded 47672 knowledge chunks from file.
[ProcessSupervisor] ❌ Synchronization Failed: ChromaConnectionError: Failed to connect to chromadb...
    at #resolveKnowledgeBaseCollection (ai/services/knowledge-base/ChromaManager.mjs:148)
[ProcessSupervisor] knowledge base sync exited with code 1.

~5 minutes elapsed between Starting knowledge base embedding (12:19:30) and the failure (12:24:33): the chroma server became unreachable during the long sync, and the collection resolve hard-failed. Operator-reported (2026-06-28), local (not cloud) orchestrator. New regression.

The Problem

Two independent gaps combine:

(1) The chroma recycle is un-coordinated with the heavy-maintenance lease. Orchestrator.poll recycles chroma when its uptime exceeds maxRuntimeMs — isChromaRecycleDue → killTask('chroma') → _chromaDefragPending → chromaDefrag. This fires purely on chroma uptime, with no check for an in-flight heavy-maintenance lease. kbSync holds that lease for the duration of a sync (syncKnowledgeBase.mjs acquires heavyMaintenanceLease). So the orchestrator can kill + defrag chroma out from under a running 5-minute, 47672-chunk sync → the next chroma call (getCollection) throws ChromaConnectionError.

(2) The KB collection resolve has no connection-error resilience. ChromaManager#resolveKnowledgeBaseCollection calls client.getCollection() and re-throws everything that isn't a "collection not found" error — no retry / reconnect / backoff. executeSilently (chromaClientPrimitives) is only a warn-suppressing sequential lock. So a brief chroma restart (recycle+defrag, crash, or any transient blip) abandons the entire sync with exit 1, instead of waiting for chroma to come back.

Likely regression trigger: the heavy-maintenance-lease fairness epic landed 2026-06-27/28 (#14205, #14273). kbSync now releases the lease at batch boundaries and resumes (so a starved heavy task interleaves), widening kbSync's wall-clock window. The un-coordinated chroma recycle/defrag now has room to fire mid-sync where it previously would not. The two gaps above are pre-existing; the lease-yield surfaced/widened the collision (this is not a defect to revert in #14273).

The Architectural Reality

• ai/daemons/orchestrator/Orchestrator.mjs (~L820-834): isChromaRecycleDue → killTask('chroma', 'max-runtime:...') + _chromaDefragPending → executeTask('chromaDefrag', ...). No heavy-maintenance-lease guard.
• ai/daemons/orchestrator/Orchestrator.mjs isChromaRecycleDue (~L777): running && maxRuntimeMs>0 && (now - lastRunAt) > maxRuntimeMs.
• ai/services/knowledge-base/ChromaManager.mjs#resolveKnowledgeBaseCollection (~L144-152): getCollection, re-throws non-not-found (incl. ChromaConnectionError) with no retry.
• ai/scripts/maintenance/syncKnowledgeBase.mjs (~L120): acquires heavyMaintenanceLease for the sync; #14273 threads the lease-yield predicate so kbSync yields + resumes.
• ai/daemons/orchestrator/services/HeavyMaintenanceLeaseService.mjs: the lease primitives the recycle should consult.

The Fix

Two parts; (a) is the primary coordination fix, (b) is defense-in-depth:

(a) Gate the chroma recycle on the heavy-maintenance lease. The recycle branch in poll (and/or isChromaRecycleDue) must defer while the heavy-maintenance lease is held — chroma recycle is opportunistic maintenance and must not preempt an active heavy sync, exactly as the lease-fairness scheme already coordinates the other heavy tasks. Recycle resumes once the lease clears.

(b) Bounded-retry the KB collection resolve on a transient ChromaConnectionError. #resolveKnowledgeBaseCollection should retry getCollection with bounded backoff on a connection error (distinct from collection-not-found / swap-in-progress, which keep their current handling), so a brief chroma restart doesn't abandon a long sync. Cap attempts + total wait; fail loud after the cap.

Decision Record impact

none — orchestrator scheduling + KB-service resilience; no ADR authority touched. (The lease config leaves live under ADR-0019's AiConfig SSOT; this ticket consumes them, it does not change the SSOT shape.)

Acceptance Criteria

• The chroma recycle (killTask('chroma') + defrag) does NOT fire while the heavy-maintenance lease is held; it defers and recycles after the lease clears. Unit coverage in the Orchestrator spec.
• #resolveKnowledgeBaseCollection retries getCollection on a transient ChromaConnectionError (bounded attempts + backoff), then fails loud; collection-not-found + swap-in-progress paths unchanged. Unit coverage in the ChromaManager spec.
• A kbSync that overlaps a recycle-eligible chroma either keeps chroma alive until the sync completes, or survives a brief chroma restart via the resolve retry — no exit 1 from a single transient blip.

Out of Scope

• The cloud orchestrator (this is the local orchestrator).
• The embedder-model eviction class (#14154, "KB-sync embedder 404s mid-sync") — a different mid-sync failure (embedder model not resident), not the chroma server connection.
• Re-architecting the chroma recycle/defrag cadence itself.

Avoided Traps

• Only adding the resolve retry (b) without (a) — would mask a self-inflicted recycle: the sync would survive but chroma still gets killed mid-sync, wasting work + risking partial state. The coordination fix (a) is primary.
• Disabling the chroma recycle — rejected: the recycle/defrag exists for bounded chroma runtime + persist-dir defrag; the fix coordinates it with the lease, not removes it.

Related

• #14273 / #14205 — the heavy-maintenance-lease fairness epic that widened the kbSync window.
• #14154 — sibling "KB-sync fails mid-sync", different layer (embedder model eviction vs chroma connection).
• #14079 — Chroma store bloat (the defrag's purpose).

Live latest-open sweep: checked latest 20 open issues at 2026-06-28; no equivalent (closest #14154 is a different mid-sync failure layer).

Handoff Retrieval Hint: "chroma recycle kbSync heavy-maintenance-lease ChromaConnectionError resolve retry"

Authored by Grace (Claude Opus 4.8, Claude Code).