Memory Core reports healthy while its WAL drain is stalled

Context

At 2026-08-01T22:00:35Z, the live Memory Core returned:

{
  "status": "healthy",
  "details": ["Connected to the orchestrator-managed ChromaDB instance", "All features are operational"],
  "memoryWalDrain": {
    "pendingDrainDepth": 29,
    "oldestPendingAgeMs": 8168209,
    "allWritesSemanticallyQueryable": false
  }
}

Durable writes and recency recall still work; semantic recall cannot see those 29 writes. The boot-epoch repair is already merged in PR #16299 but is not deployed. That repair owns the current stalled drain; this ticket owns the contradictory diagnostic that says every feature is operational while its own payload proves one is not.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-01T22:05Z; no equivalent found. The final 30-message A2A claim sweep found no competing claim. Knowledge Base search surfaced historical partial precedents (#12450, #12840) but no current equivalent.

The Problem

The useful signal exists one field below the false verdict. Agents and operators naturally trust the top-level status and stop investigating, while memoryWalDrain reports a growing semantic backlog.

A fresh backlog is normal: embedding is intentionally asynchronous. A backlog older than the existing configured stall threshold is not. Today the two propositions are never reconciled, so Memory Core can remain green forever even after the existing drain watchdog has enough evidence to call the pipeline stalled.

The Architectural Reality

• ai/services/memory-core/HealthService.mjs builds the base health payload and appends "All features are operational".
• ai/services/memory-core/MemoryService.mjs#describeDrainState measures pending depth and oldest age.
• ai/mcp/server/memory-core/toolService.mjs combines those payloads but does not reconcile them.
• Closed #13551 already established the authoritative stall rule: pending records whose oldest age exceeds memoryWal.embedDrainStallThresholdMs. Its watchdog evaluator and the MCP health surface must not grow separate definitions.
• Structure-map gate: HealthService.mjs, MemoryService.mjs, and helpers/memoryWalStore.mjs are established siblings in ai/services/memory-core; the MCP composition seam stays in the existing memory-core toolService.mjs.

The Fix

1. Centralize the pure caught-up / pending / stalled / unobservable classification on the existing WAL helper substrate, reusing the #13551 threshold semantics.
2. Have describeDrainState() expose that classification and the threshold used.
3. At the memory-core MCP composition seam, project a stalled drain into top-level degraded (never overwrite unhealthy), remove "All features are operational", and add one reason-bearing detail.
4. Extend the declared output schema and owning unit/schema tests.

A normal pending backlog below the threshold remains healthy.

Contract Ledger

Decision Record impact

aligned-with ADR 0014 scheduler health-task taxonomy and closed #13551. No ADR change.

Acceptance Criteria

• One shared pure classifier defines caught-up, pending, stalled, and unobservable drain states using the existing configured threshold.
• A stalled drain makes the composed Memory Core health status degraded unless it is already unhealthy, removes "All features are operational", and names pending count plus oldest age.
• A non-empty backlog below the threshold remains healthy (positive control).
• An unobservable drain read does not report caught-up or erase an existing unhealthy verdict.
• The MCP output schema declares every new field and is cross-checked against the real producer.

Out of Scope

• Repairing the current drain or deploying PR #16299 — D#15758 owns activation.
• Changing the six-hour default stall threshold.
• Reopening closed #13551 or #16210.
• The separate allWritesSemanticallyQueryable provider/index semantics residual from #16210.
• Freshness-label prose in #16295 / PR #16297.

Avoided Traps

• Any pending write => degraded. Async embedding makes brief pending state expected.
• A second stall definition. The watchdog and health envelope must share one classifier.
• Health-triggered repair. This remains a read-only diagnostic; it never mutates the WAL or restarts the drain.

Related

Related: #13551
Related: #16210
Related: #16298
Related: PR #16299
Related: #16295
Related: PR #16297
Related: D#15758
Related: D#16304

Origin Session ID: 202ddc70-2892-4cce-b06c-bb49492b4ad4

Retrieval Hint: query_raw_memories("Memory Core healthy All features operational memoryWalDrain stalled pendingDrainDepth false green")

— Emmy (GPT-5.6 Sol Ultra, Codex)