LearnNewsExamplesServices
Frontmatter
id14124
titleRead-only Memory Core diagnostics gated by the embedding-write canary — a catch-22 when the embed provider is slow
stateClosed
labels
enhancementaiarchitecturemodel-experience
assigneesneo-opus-ada
createdAtJun 26, 2026, 6:49 PM
updatedAtJun 27, 2026, 2:11 AM
githubUrlhttps://github.com/neomjs/neo/issues/14124
authorneo-opus-vega
commentsCount4
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 27, 2026, 2:11 AM

Read-only Memory Core diagnostics gated by the embedding-write canary — a catch-22 when the embed provider is slow

Closed v13.1.0/archive-v13-1-0-chunk-7 enhancementaiarchitecturemodel-experience
neo-opus-vega
neo-opus-vega commented on Jun 26, 2026, 6:49 PM

The friction (hit live, 2026-06-26)

During the v13.1 rem-consolidation-stall investigation, get_rem_pipeline_state (a read-only diagnostic) failed with:

Cannot execute get_rem_pipeline_state: Memory Core is not fully operational: Embedding write canary timed out after 5000ms

The embedding provider (openAiCompatible http://127.0.0.1:1234, text-embedding-qwen3-embedding-8b, 4096d) was slow-but-alivehealthcheck with a 12s canary returned healthy (22,637 memories, 1,411 summaries, Chroma connected, backups current, all features operational). So the read-only diagnostic that would let an agent see the slow-embed degradation is itself disabled by that degradation — a catch-22. I had to work around it (healthcheck + a bumped canary timeout) to root-cause the stall.

The Problem

The Memory Core's "fully operational" gate (the embedding-write canary) is a correct fail-safe for write operations — don't write to a non-writable store. But it also gates read-only diagnostics (get_rem_pipeline_state, and likely other read-only state projections). When the embed provider is slow (write-canary > the 5s default), the agent loses the very diagnostics needed to root-cause the slowness. This blunts the immune system at exactly the moment it is needed — a degradation that should be observable becomes un-observable through the canonical tool.

The Fix (proposed)

  • Read-only diagnostics degrade gracefully: return their data annotated with a degraded / embeddingCanary: 'slow'|'failed' warning instead of hard-failing.
  • The write-canary gate stays for write operations (correct fail-safe).
  • Distinguish read-only-diagnostic from write-op at the operational-gate choke point.

Out of Scope

  • The embed-provider perf itself (operator-infra: provider/model speed, LM Studio load).
  • Changing the write-canary gate for write operations.

Avoided Traps

  • Don't just bump the canary timeout globally — that masks real slowness for writes (a genuine fail-safe) and doesn't fix the catch-22 for diagnostics.
  • Don't remove the gate — it is a correct write fail-safe.

Related

  • #14039 (v13.1 immune system — this friction surfaced during a live rem-consolidation-stall diagnosis the immune system correctly flagged), #12065 (REM/Sandman pipeline), #13435 (sibling pattern: decouple a healthcheck from an over-aggressive gate).

Live latest-open sweep: scanned the 40 latest open issues — no existing ticket for the canary-gates-read-only-diagnostics catch-22 (#13435 is the gitlab-pat-gate analog, not this). A2A: no in-flight claim.

Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8). Origin Session 16bbea8d-8bc9-4dad-8e1c-8e3b2cd861a3.

tobiu referenced in commit 741f27e - "fix(ai): exempt read-only Memory Core diagnostics from the embed-canary health gate (#14124) (#14162) on Jun 27, 2026, 2:11 AM
tobiu closed this issue on Jun 27, 2026, 2:11 AM