LearnNewsExamplesServices
Frontmatter
id12628
titleStorageRouter swallows ChromaDB query failures as silent empty results
stateClosed
labels
bugaiarchitecturemodel-experience
assigneesneo-opus-grace
createdAtJun 6, 2026, 12:17 PM
updatedAtJun 6, 2026, 5:02 PM
githubUrlhttps://github.com/neomjs/neo/issues/12628
authorneo-opus-grace
commentsCount0
parentIssue12450
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 6, 2026, 5:02 PM

StorageRouter swallows ChromaDB query failures as silent empty results

Closed v13.0.0/archive-v13-0-0-chunk-16 bugaiarchitecturemodel-experience
neo-opus-grace
neo-opus-grace commented on Jun 6, 2026, 12:17 PM

Context

Downstream observability leaf of #12450. The parent's Layer-2 finding — that StorageRouter.injectQueryReRanker converts any Pass-1 ChromaDB throw into a clean empty result — was Verify-Before-Assert'd live this session: the swallow is unchanged at ai/services/memory-core/managers/StorageRouter.mjs:79-82:

} catch (e) {
    logger.warn(`[StorageRouter] Pass 1 semantic retrieval failed, falling back to empty result: ${e.message}`);
    return {ids: [[]], distances: [[]], metadatas: [[]], documents: undefined};
}

This leaf carries the codeable observability half of #12450 so it can ship autonomously. The parent stays open for the Layer-1 live HNSW-segment repair — a live-DB action that is ops/operator-coordinated, not an agent-code lane.

Live latest-open sweep: checked the latest 25 open issues at creation; no equivalent observability leaf found (parent #12450 carries the combined corruption+observability scope; this narrows the codeable half).

The Problem

A populated-but-corrupt collection (parent #12450: neo-agent-sessions, 68/1018 vectors reachable, get(embeddings) throws Error finding id) reads to every agent as {count:0, results:[]}indistinguishable from a genuine no-match. That is exactly why the corruption was mis-classified as a "transient glitch" for days. A resilience fallback that erases the failure signal is worse than the crash it replaced: it converts a loud, diagnosable error into a silent, multi-day-latent data-availability outage.

The swallow itself (graceful non-crash) is correct; the missing upward observability is the bug.

The Architectural Reality

  • ai/services/memory-core/managers/StorageRouter.mjs:65-82 injectQueryReRanker() — the Pass-1 try/catch, injected on BOTH memory and summary collections (getMemoryCollection/getSummaryCollection). The success path already stamps a _reRanked: true marker (line 149); the fallback should symmetrically stamp a _degraded marker.
  • ai/services/memory-core/SummaryService.mjs querySummaries() + ai/services/memory-core/MemoryService.mjs queryMemories() — the tool-facing callers that currently collapse the empty result into {count:0, results:[]}. They must detect the degraded marker and surface a distinguishable envelope.
  • ai/services/memory-core/HealthService.mjs:864-873 — reports count per collection with no query-path probe, so a count()-able-but-not-query()-able collection passes as status:healthy.

The Fix

Non-throwing, decoupled from the Layer-1 data repair:

  1. injectQueryReRanker (StorageRouter): logger.warnlogger.error; tag the Error finding id corruption signature distinctly + name the collectionType; stamp a _degraded: true + _degradedReason marker on the fallback result instead of a bare clean-empty. Still returns (no crash; Pass-2 resilience preserved).
  2. querySummaries / queryMemories: when the re-ranked result carries _degraded, surface a structured degraded envelope (e.g. {degraded: true, code: 'QUERY_PATH_DEGRADED', collection, message}) instead of a silent {count:0, results:[]} — so "0 genuine matches" and "query path failed" are distinguishable to the caller.
  3. HealthService query-canary: add a lightweight per-collection query (or reachable-vs-record-count divergence) probe; degrade status + name the failing collection when a populated collection's query() throws or returns near-zero reachable vectors.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
StorageRouter.injectQueryReRanker Pass-1 catch ai/services/memory-core/managers/StorageRouter.mjs:79-82 logger.error + corruption-signature tag + _degraded/_degradedReason marker on the fallback result non-throwing degraded result (Pass-2 path preserved) JSDoc on injectQueryReRanker the live swallow block (V-B-A'd this session) + parent #12450's 14× log evidence
query_summaries / query_raw_memories tool response SummaryService.querySummaries / MemoryService.queryMemories degraded envelope (code:'QUERY_PATH_DEGRADED') when _degraded is set — never a silent {count:0} structured degraded result consumed by callers ai/mcp/server/memory-core/openapi.yaml parent #12450 probe-asymmetry table
healthcheck collection report ai/services/memory-core/HealthService.mjs:864-873 per-collection query-canary; degrade status + structured field on a populated-but-unqueryable collection status degraded naming the failing collection openapi.yaml healthcheck schema count:1018 reported healthy while 68/1018 reachable

Decision Record impact

none — embodies the general "no silent fallback for a real failure" principle; no ADR asserted (consistent with parent #12450).

Acceptance Criteria

  • injectQueryReRanker stamps a _degraded marker (+ logger.error + corruption-signature tag naming the collection) on a Pass-1 throw instead of returning a bare clean-empty result. Non-throwing.
  • query_summaries / query_raw_memories surface a structured degraded envelope (distinguishable from a genuine no-match) when the underlying query path is degraded.
  • healthcheck runs a per-collection query-canary (or reachable-vs-record divergence) and reports degraded + a structured field naming the collection when a populated collection's query path throws / has near-zero reachable vectors.
  • Regression test: a collection whose query() throws (Error finding id or any ChromaDB error) yields a surfaced degraded signal from query_summaries/query_raw_memories, not a silent {count:0}; healthcheck reflects degraded for that collection.

Out of Scope

  • Layer-1 live HNSW-segment repair (re-embed / rebuild / restore neo-agent-sessions) — stays on parent #12450; a live-DB ops action, operator/ops-coordinated, not an agent-code lane.
  • Changing the embedding provider/model or vector dimensions — V-B-A-falsified on #12450 (stored == query == 4096; the sibling collection is healthy on the same embedder).
  • The access-policy (team vs private) summary-read fix — already delivered via #12473 / #12470 (the Layer-3 leaf).

Avoided Traps

  • "Just repair the data and close it" — leaves the silent-failure mode that hid the corruption for days; it recurs invisibly. The observability fix is the load-bearing, durable half.
  • Re-throwing instead of marking — would crash the query pipeline the original swallow exists to protect (Pass-2 re-ranking resilience). The fix marks + surfaces; it does not re-raise.

Related

  • #12450 — parent (corruption + observability); this leaf delivers its Layer-2 observability ACs, parent stays open for Layer-1 repair.
  • #9959 — origin of the Pass-1 swallow (its Bug-1 resilience prescription); being resolved separately (Bug-2 active-session leak) via #12605.
  • #12473 / #12470 — the Layer-3 access-policy leaf (merged).
  • #12133 / #11924 — sibling Error finding id ChromaDB-corruption signatures.

_degraded marker shape is a Tier-2 local/reversible decision taken at authoring time (adjustable in review); not blocked on operator input.

Origin Session ID: 0f6d0fa0-327f-42ec-b970-e32f388699b4

Retrieval Hint: "StorageRouter injectQueryReRanker swallow degraded marker query_summaries observability Error finding id"

Authored by Claude Opus 4.8 (Claude Code) as @neo-opus-grace.

tobiu referenced in commit ef94a4b - "fix(memory-core): surface swallowed query failures as a degraded result (#12628) (#12629) on Jun 6, 2026, 5:02 PM
tobiu closed this issue on Jun 6, 2026, 5:02 PM