The friction (hit live, 2026-06-26)
During the v13.1 rem-consolidation-stall investigation, get_rem_pipeline_state (a read-only diagnostic) failed with:
Cannot execute get_rem_pipeline_state: Memory Core is not fully operational: Embedding write canary timed out after 5000ms
The embedding provider (openAiCompatible http://127.0.0.1:1234, text-embedding-qwen3-embedding-8b, 4096d) was slow-but-alive — healthcheck with a 12s canary returned healthy (22,637 memories, 1,411 summaries, Chroma connected, backups current, all features operational). So the read-only diagnostic that would let an agent see the slow-embed degradation is itself disabled by that degradation — a catch-22. I had to work around it (healthcheck + a bumped canary timeout) to root-cause the stall.
The Problem
The Memory Core's "fully operational" gate (the embedding-write canary) is a correct fail-safe for write operations — don't write to a non-writable store. But it also gates read-only diagnostics (get_rem_pipeline_state, and likely other read-only state projections). When the embed provider is slow (write-canary > the 5s default), the agent loses the very diagnostics needed to root-cause the slowness. This blunts the immune system at exactly the moment it is needed — a degradation that should be observable becomes un-observable through the canonical tool.
The Fix (proposed)
- Read-only diagnostics degrade gracefully: return their data annotated with a
degraded / embeddingCanary: 'slow'|'failed' warning instead of hard-failing.
- The write-canary gate stays for write operations (correct fail-safe).
- Distinguish read-only-diagnostic from write-op at the operational-gate choke point.
Out of Scope
- The embed-provider perf itself (operator-infra: provider/model speed, LM Studio load).
- Changing the write-canary gate for write operations.
Avoided Traps
- Don't just bump the canary timeout globally — that masks real slowness for writes (a genuine fail-safe) and doesn't fix the catch-22 for diagnostics.
- Don't remove the gate — it is a correct write fail-safe.
Related
- #14039 (v13.1 immune system — this friction surfaced during a live rem-consolidation-stall diagnosis the immune system correctly flagged), #12065 (REM/Sandman pipeline), #13435 (sibling pattern: decouple a healthcheck from an over-aggressive gate).
Live latest-open sweep: scanned the 40 latest open issues — no existing ticket for the canary-gates-read-only-diagnostics catch-22 (#13435 is the gitlab-pat-gate analog, not this). A2A: no in-flight claim.
Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8). Origin Session 16bbea8d-8bc9-4dad-8e1c-8e3b2cd861a3.
The friction (hit live, 2026-06-26)
During the v13.1 rem-consolidation-stall investigation,
get_rem_pipeline_state(a read-only diagnostic) failed with:The embedding provider (
openAiCompatiblehttp://127.0.0.1:1234,text-embedding-qwen3-embedding-8b, 4096d) was slow-but-alive —healthcheckwith a 12s canary returned healthy (22,637 memories, 1,411 summaries, Chroma connected, backups current, all features operational). So the read-only diagnostic that would let an agent see the slow-embed degradation is itself disabled by that degradation — a catch-22. I had to work around it (healthcheck + a bumped canary timeout) to root-cause the stall.The Problem
The Memory Core's "fully operational" gate (the embedding-write canary) is a correct fail-safe for write operations — don't write to a non-writable store. But it also gates read-only diagnostics (
get_rem_pipeline_state, and likely other read-only state projections). When the embed provider is slow (write-canary > the 5s default), the agent loses the very diagnostics needed to root-cause the slowness. This blunts the immune system at exactly the moment it is needed — a degradation that should be observable becomes un-observable through the canonical tool.The Fix (proposed)
degraded/embeddingCanary: 'slow'|'failed'warning instead of hard-failing.Out of Scope
Avoided Traps
Related
Live latest-open sweep: scanned the 40 latest open issues — no existing ticket for the canary-gates-read-only-diagnostics catch-22 (#13435 is the gitlab-pat-gate analog, not this). A2A: no in-flight claim.
Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8). Origin Session 16bbea8d-8bc9-4dad-8e1c-8e3b2cd861a3.