Bound Memory Core health probes when providers stall

Context

A containerized Memory Core deployment accepted PAT authentication and WAL-local add_memory writes, but several read/diagnostic calls left the MCP Streamable HTTP response open until the client timed out. The observed split was specific:

• add_memory returned quickly, which now proves WAL acceptance, not vector persistence.
• query_recent_turns returned quickly with a fail-closed identity scope, proving the transport/auth layer was not globally hung.
• query_raw_memories, resume_session, and get_rem_pipeline_state sent HTTP/SSE headers but never produced a JSON-RPC result event before the caller timeout.
• Local development Memory Core healthcheck remained healthy and fast with a large memory collection, so collection size alone is not the reproducer.

This points to a provider/vector diagnostic path that can await forever. Healthcheck should degrade or fail boundedly; it must not hang the MCP request.

Release classification: post-release deployment hardening / model-experience resilience. Boardless.

Live latest-open sweep: checked the latest 20 open GitHub issues/PRs on 2026-06-18; no equivalent found. Closest live issues were #12450 (query-path corruption/empty-result observability), #13435 (loopback healthcheck auth coupling), and closed #12978 (non-embedding read health-gate exemptions); none covers unbounded downstream health/diagnostic probes.

A2A in-flight sweep: checked recent inbox messages with list_messages({status: 'all', limit: 30}); no overlapping [lane-claim] / [lane-intent] surfaced.

Knowledge Base ticket sweep: asked for open tickets about Memory Core healthcheck hanging, unbounded timeout, embedding canary, Chroma, and MCP healthcheck; result found adjacent closed/partial tickets but no equivalent open ticket.

The Problem

The Memory Core health/diagnostic surfaces catch thrown failures, but several awaited downstream promises are not bounded. If Chroma, Ollama/openAI-compatible embeddings, or a REM axis stalls instead of throwing, the MCP tool request can stay open indefinitely.

This is worse than a degraded response:

• Non-exempt tools are health-gated through ensureHealthy(), so a stalled health probe prevents query_raw_memories from returning the intended health-gate rejection.
• Container healthchecks rely on mcpHealthcheck.mjs, but that client call has no internal timeout of its own.
• Exempt read tools can still hang if their own Chroma metadata reads never resolve before their fallback logic runs.

Prior-art sweep: a June 8 miniSummary backfill incident diagnosed the same failure class: a model call hung rather than threw, so fail-soft code never executed. Memory Core already has ai/services/memory-core/helpers/withTimeout.mjs for this exact shape.

Architectural Reality

Current unbounded paths include:

• ai/services/memory-core/HealthService.mjs buildEmbeddingWriteCanaryBlock() awaits embedText() directly.
• HealthService.#checkDatabaseConnections() awaits Chroma ready/connect directly.
• HealthService.#checkCollections() awaits StorageRouter collection resolution and collection count() directly.
• HealthService.buildRemPipelineState() resolves multiple Chroma/graph/topology axes in Promise.all() without bounding individual axes.
• ai/scripts/diagnostics/mcpHealthcheck.mjs awaits MCP client connect() and callTool({name: 'healthcheck'}) directly.
• ai/services/memory-core/SessionService.mjs#validateSessionForResume() awaits StorageRouter.getMemoryCollection() and collection.get() before graph fallback.
• ai/services/memory-core/MemoryService.mjs#listMemories() awaits StorageRouter.getMemoryCollection() and collection.get() directly.

Existing test coverage is green for thrown failures, but it does not cover promises that never resolve.

The Fix

Reuse ai/services/memory-core/helpers/withTimeout.mjs instead of adding a new timing primitive. Bound the downstream probe/read calls that can block healthcheck and diagnostic tools, then surface the timeout as structured degraded/error information rather than leaving the request open.

Candidate implementation shape:

• Add small health/diagnostic timeout constants or config leaves aligned with the container healthcheck budget.
• Wrap healthcheck Chroma ready/connect/count probes and the embedding write canary.
• Wrap REM axes individually so one stalled axis degrades that axis instead of hanging the whole tool.
• Wrap mcpHealthcheck client connect/tool-call so the script exits with a clear timeout failure before Docker or the harness kills it opaquely.
• Wrap resume_session / get_session_memories Chroma metadata reads so fallbacks/empty results can resolve.

Contract Ledger Matrix

Decision Record impact

none. This aligns with existing Memory Core resilience posture and the existing shared withTimeout helper; no ADR authority change is needed.

Acceptance Criteria

• HealthService.healthcheck() resolves within a bounded unit-test interval when the embedding write canary promise never resolves, and reports degraded with the timeout named.
• HealthService.healthcheck() resolves within a bounded unit-test interval when Chroma collection resolution or count() never resolves, and reports a degraded/error collection state instead of hanging.
• buildRemPipelineState() returns when one REM axis never resolves, with that axis marked degraded/error and other axes preserved.
• mcpHealthcheck.runHealthcheck() rejects with a clear timeout when connect() or callTool() never resolves, and still closes the client when applicable.
• resume_session and get_session_memories do not hang on stalled Chroma metadata .get() calls; their fallback/empty responses resolve.
• Focused unit tests cover the never-resolving-promise class, not just thrown errors.

Out of Scope

• Fixing the root cause of any specific provider/Chroma/Ollama stall.
• Changing the public auth model for in-container healthchecks (#13435).
• Repairing corrupt vector collections or adding query-reachability canaries beyond the timeout guard (#12450).
• Re-opening the already-closed health-gate exemption scope from #12978; query_raw_memories remains gated because it embeds the query.

Avoided Traps

• Treating add_memory success as proof of vector persistence. In the WAL-first design it only proves durable write acceptance.
• Catching only thrown errors. The production-class failure is a promise that never resolves.
• Exempting semantic query tools from the health gate. That converts a bounded gate failure into an embed-time stall; this ticket instead bounds the gate and probe paths.

Related

• #12450 — adjacent query-path corruption / silent empty-result observability.
• #12978 — closed non-embedding read exemption audit.
• #13435 — adjacent loopback healthcheck auth coupling.
• Prior Memory Core incident: miniSummary backfill hung on an unbounded model call; reuse the withTimeout pattern already present in Memory Core.

Handoff Retrieval Hints

• Retrieval Hint: "Memory Core healthcheck hangs embedding canary Chroma timeout withTimeout"
• Retrieval Hint: "add_memory WAL accepted query_raw_memories resume_session SSE headers no result"
• Origin Session ID: 4ce60429-2986-4543-be2d-741957c75b6c

Authored by GPT-5 (Codex Desktop) / @neo-gpt.