LearnNewsExamplesServices
Frontmatter
id14181
titleMemory Core embed-write health canary (5s default) false-negatives a cold 8b embedder — raise to ≥30s
stateClosed
labels
bugai
assigneesneo-opus-ada
createdAtJun 27, 2026, 3:18 AM
updatedAtJun 27, 2026, 8:50 AM
githubUrlhttps://github.com/neomjs/neo/issues/14181
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 27, 2026, 8:50 AM

Memory Core embed-write health canary (5s default) false-negatives a cold 8b embedder — raise to ≥30s

neo-opus-ada
neo-opus-ada commented on Jun 27, 2026, 3:18 AM

Friction (operator-surfaced 2026-06-27)

MC reports not fully operational: Embedding write canary timed out after 5000ms — blocking every embedding-dependent tool (query_raw_memories, add_memory) while non-embedding paths (add_message, mark_read) keep working. Root-caused live: the openAiCompatible embedder (LM Studio :1234, text-embedding-qwen3-embedding-8b) is up, but each embed runs 11–19s because the 8b model VRAM-thrashes under the co-resident gemma-4 chat models and cold-reloads per request (the #14154 co-eviction, now measured). Against a 5000ms canary, that's a false-negative: MC health goes intermittent (green only when a peer catches the model resident).

The mismatch

healthcheck.embeddingWriteCanaryTimeoutMs defaults to 5000ms (ai/mcp/server/memory-core/config.template.mjs:318), but the embed operation itself budgets 300000ms (embeddingTimeoutMs, ai/config.template.mjs:172) — a 60× gap. The health gate is 60× stricter than the operation it gates, so it declares a healthy-but-slow embedder dead.

Origin: d8cf634da6 — Euclid's fix(memory-core): bound health probes (#13458) (2026-06-18). The intent (bound probes so they can't hang) is correct and must be preserved; only the value overshot — it never accounted for an 8b embedder's cold-load latency.

Fix

Raise the embeddingWriteCanaryTimeoutMs default 5000 → 30000 in config.template.mjs (the SSOT) + the mirrored openapi.yaml default, with a rationale comment (must tolerate an 8b cold-load of 11–19s; still a bound, just a realistic one). Stays env-overridable via NEO_MEMORY_HEALTHCHECK_EMBEDDING_WRITE_CANARY_TIMEOUT_MS. Does NOT undo #13458's bounding — 30s is still a bound, not removal.

Acceptance Criteria

  • embeddingWriteCanaryTimeoutMs default is 30000ms in config.template.mjs + openapi.yaml (lint-config-template-ssot stays green).
  • A cold embed (11–19s) no longer trips the canary → query_raw_memories / add_memory survive a cold-but-working embedder.
  • The bound is preserved (a genuinely-hung embedder still fails, just at 30s not 5s).

Related

#14154 (the VRAM co-eviction this canary false-negatives on — the deeper root, separate lane: pin embedder / reduce VRAM pressure), #13458 / #13459 (the bound-health-probes origin), #14124 (the embed-canary health-gate + read-only exemptions).


🤖 Authored by Ada (@neo-opus-ada · Claude Opus 4.8, Claude Code) · origin session f2c722bf-9fb0-4925-8fbc-a9a0788f459c. Surfaced by @tobiu; friction → gold.

tobiu referenced in commit 2e79911 - "fix(memory-core): raise embed-write canary timeout 5s->30s for a cold 8b embedder (#14181) (#14182) on Jun 27, 2026, 8:50 AM
tobiu closed this issue on Jun 27, 2026, 8:50 AM