Recover stale Chroma handles after daemon recycle

Context

A local Neo investigation on 2026-06-18 found that the Knowledge Base corpus was not physically lost, but a long-lived MCP server reported stale collection state after the orchestrator recycled Chroma.

Observed evidence:

• Live Chroma SQLite reported neo-knowledge-base|30466 rows.
• The newest JSONL backup under .neo-ai-data/backups/backup-2026-06-18T07-42-28.523Z/ also contained 30466 KB rows.
• A fresh source-level KB HealthService.healthcheck({freshObservability:true}) saw status: healthy and count: 30466.
• The attached long-lived KB MCP process still reported an unhealthy collections.knowledgeBase:null / resource could not be found surface.
• Process timing matched the stale-handle theory: the attached KB MCP process started before the orchestrator recycled Chroma, while the Chroma daemon and KB defrag/reload happened later.

Live latest-open sweep: checked the latest 20 open issues/PRs via gh api --method GET repos/neomjs/neo/issues -f state=open -f per_page=20 on 2026-06-18; no equivalent ticket found. Adjacent but not duplicate: #13289 tracks stale MCP/bridge code relative to checkout state, not stale Chroma collection handles after daemon recycle.

A2A in-flight claim sweep: checked latest 30 messages with list_messages({status:'all', limit:30}); no overlapping [lane-claim] / [lane-intent] found.

Release classification: post-release / operational-hardening follow-up; boardless unless the operator promotes it to release-blocking.

The Problem

Long-lived MCP server processes can keep collection objects or collection-resolution promises that were resolved against an older Chroma daemon generation. When the orchestrator owns Chroma lifecycle and performs a max-runtime recycle plus KB defrag/reload, the old MCP process may keep serving stale collection state even after the live store has recovered and a fresh process sees the correct corpus.

This is not limited to the Knowledge Base surface. Memory Core has the same underlying class of risk because it caches Chroma collection promises for the memory, session-summary, and graph collections. That can make healthcheck counts, query paths, or tool availability diverge from the actual live Chroma store after a Chroma restart/recycle.

The Architectural Reality

Relevant current paths:

• ai/daemons/orchestrator/Orchestrator.mjs recycles Chroma when max runtime is exceeded, then runs the one-shot chromaDefrag task after Chroma is connection-ready.
• ai/daemons/orchestrator/taskDefinitions.mjs defines chromaDefrag as ai/scripts/maintenance/defragChromaDB.mjs --target knowledge-base.
• ai/scripts/maintenance/defragChromaDB.mjs currently buffers data, deletes the canonical collection, recreates it, then reloads batches. The observed run completed, but this creates a collection-generation boundary that old collection objects may not survive.
• ai/services/knowledge-base/ChromaManager.mjs now has invalidateKnowledgeBaseCollectionCache() and a not-found classifier. KB healthcheck already retries a collection-count failure once after invalidation.
• ai/services/memory-core/managers/ChromaManager.mjs caches _memoryCollectionPromise, _summaryCollectionPromise, and _graphCollectionPromise for the singleton lifetime, but does not expose the same production invalidation/retry shape.
• ai/services/memory-core/HealthService.mjs refreshes request-facing counts through StorageRouter.getMemoryCollection() / StorageRouter.getSummaryCollection(), which still delegates to the cached singleton collection promises.
• test/playwright/unit/ai/services/memory-core/util.mjs already proves the required reset semantics in tests: clearing MC lifecycle state also clears Chroma collection promises and handles. The production path needs an analogous, targeted recovery primitive.

The Fix

Implement stale Chroma collection-handle recovery across KB and MC:

• Add a Memory Core collection-cache invalidation method that clears memory, summary, and graph collection promises/handles without requiring a full test-only lifecycle reset.
• Add a shared or mirrored Chroma stale-handle classifier for operation-level errors such as resource could not be found, not found, Chroma 404, or ChromaNotFoundError.
• Update Memory Core healthcheck count paths to invalidate and retry once when a cached collection handle fails with a stale-handle/not-found signature.
• Audit Memory Core read/write tool paths that hold a collection object only long enough for an operation; ensure a stale-handle failure does not get collapsed into misleading empty counts or generic unhealthy state when a re-resolve would recover.
• Keep Knowledge Base behavior aligned with the same shared classification shape so KB and MC do not drift.

Contract Ledger Matrix

Decision Record impact

none. This is aligned with the orchestrator-owned Chroma lifecycle and the existing runtime-freshness direction; it does not amend an ADR.

Acceptance Criteria

• Memory Core exposes a production-safe collection-cache invalidation primitive covering memory, summary, and graph collections.
• Memory Core healthcheck retries once after invalidating cached collection handles when memory/session count operations fail with stale-handle / not-found signatures.
• Knowledge Base and Memory Core use the same stale-handle classification semantics, either through a shared helper or intentionally mirrored predicates with tests.
• Unit coverage simulates a stale cached collection object whose count() fails, then verifies the second resolution returns the live count.
• Unit coverage includes both MC memory and MC summary collection paths.
• Manual validation confirms a long-lived MCP server does not keep reporting missing/zero collection counts after Chroma is recycled and live Chroma contains the data.

Out of Scope

• Repairing the malformed Chroma FTS5 index reported by pragma quick_check.
• Redesigning defragChromaDB.mjs to remove the delete/recreate crash window.
• Restoring or rebuilding genuinely missing/corrupt Chroma collections.
• Changing orchestrator max-runtime recycle policy.

Avoided Traps

• Do not solve this by only restarting MCP processes manually. Restart is a useful operator workaround, but the defect is that a healthy live Chroma store can be misreported by a stale long-lived process.
• Do not collapse stale-handle failures into count:0. Zero rows and stale collection resolution are different operational states.
• Do not use OS PID alone as source of truth. A failed collection operation plus successful re-resolve is the direct runtime proof.

Related

Related: #13289

Origin Session ID

Origin Session ID: 4ce60429-2986-4543-be2d-741957c75b6c

Handoff Retrieval Hints

• query_raw_memories: Chroma restart stale collection handle affects Memory Core and Knowledge Base MCP healthcheck counts
• query_raw_memories: long lived MCP server stale Chroma client after orchestrator recycle Memory Core sessions summaries collection
• Local evidence anchors: orchestrator log around 2026-06-18T06:33:59Z through 2026-06-18T06:59:33Z; latest KB backup backup-2026-06-18T07-42-28.523Z; live Chroma SQLite count neo-knowledge-base|30466.