LearnNewsExamplesServices
Frontmatter
id12737
titleReturn degraded KB refs when ask synthesis provider fails
stateClosed
labels
bugairegressionmodel-experience
assigneesneo-gpt
createdAtJun 8, 2026, 7:32 AM
updatedAtJun 8, 2026, 10:52 AM
githubUrlhttps://github.com/neomjs/neo/issues/12737
authorneo-gpt
commentsCount0
parentIssue12696
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 8, 2026, 10:52 AM

Return degraded KB refs when ask synthesis provider fails

Closed v13.0.0/archive-v13-0-0-chunk-17 bugairegressionmodel-experience
neo-gpt
neo-gpt commented on Jun 8, 2026, 7:32 AM

Context

During v13 release work on 2026-06-08, ask_knowledge_base hard-failed while investigating the sync_all release-cache lane:

[GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent: [429 Too Many Requests]

The failure happened after the tool had already entered the RAG path, but the caller received only the provider exception. That matches the operator-reported MX friction that the ask tool was "WAY better before" and now regresses under current v13 load.

Live latest-open sweep: checked the latest 20 open issues at 2026-06-08T05:31Z (including author, title, labels, URL); no equivalent ticket. Closest current tickets are #12715 (lexical-rescue I/O), #12719 (candidate weighting), #12450 (Memory Core query_summaries Chroma error), and #12713 (miniSummary provider hang), but none cover SearchService.ask losing already-retrieved KB references when synthesis fails.

Duplicate sweeps:

  • GitHub open issue search: ask_knowledge_base quota fallback degraded synthesis -> no results.
  • GitHub open PR search: ask_knowledge_base quota fallback degraded synthesis -> no results.
  • KB ticket semantic sweep via query_documents(type='ticket') surfaced older generic KB/search issues, not an equivalent synthesis-provider fallback ticket.
  • Exact corpus sweep over resources/content/issues, resources/content/archive/issues, and resources/content/discussions surfaced related KB degradation history (#12450, #12133, #11924) and the ask-vs-query dominance note, but no issue that pins the synthesis quota/error fallback contract.

The Problem

ai/services/knowledge-base/SearchService.mjs retrieves ranked references before invoking the synthesis model:

  1. QueryService.queryDocuments({query, type, limit, includeMetadata: true})
  2. Build references
  3. Hydrate context
  4. Call this.model.generateContent(prompt)

When step 4 fails with provider quota, transient provider outage, or missing/invalid generation model, the current implementation throws. The MCP caller loses the ranked KB evidence that was already available, forcing agents into a worse fallback path and hiding whether semantic search itself was healthy.

This is especially costly during v13 because agents are leaning on the KB for release-note evidence, code-location discovery, and friction triage under high provider load.

The Architectural Reality

  • Owning service: ai/services/knowledge-base/SearchService.mjs.
  • Retrieval source of authority: ai/services/knowledge-base/QueryService.mjs remains the ranked reference provider.
  • Current tests live in test/playwright/unit/ai/services/knowledge-base/SearchService.spec.mjs and already stub QueryService.queryDocuments plus SearchService.model.generateContent, making this regression cheap to cover without live Gemini.
  • MCP consumer contract: ask_knowledge_base is documented as the default Neo knowledge query because it returns a synthesized answer plus references. Under synthesis failure, it should degrade to references-plus-status, not erase the retrieval payload.

The Fix

Update SearchService.ask so synthesis-provider failures return a degraded response object instead of throwing after retrieval succeeds.

Expected shape:

  • Keep returning references built from QueryService.queryDocuments.
  • Return an answer that explicitly states synthesis failed and instructs the caller to use the references directly.
  • Add a machine-readable degraded marker, e.g. degraded: true, plus a bounded error / reason string that does not leak credentials.
  • Preserve true retrieval failures as failures; this ticket only covers the post-retrieval synthesis path.
  • Add focused unit coverage in SearchService.spec.mjs for generateContent rejection after references exist.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
SearchService.ask() return payload Existing SearchService.ask references contract + MCP ask_knowledge_base tool description If retrieval succeeds but synthesis fails, return {answer, references, degraded: true, error} instead of throwing If retrieval itself fails, preserve the existing thrown failure path JSDoc / inline comments if return shape is widened Unit test stubs generateContent rejection and asserts references survive
ask_knowledge_base MCP consumer behavior Tool description says ask returns synthesized answer plus refs and is the default Neo query tool Agents receive ranked refs even when Gemini quota/provider is exhausted Caller can still switch to query_documents manually if refs are insufficient No OpenAPI schema change unless the tool has a strict output schema Local unit test plus live manual recheck when provider recovers
Error disclosure AGENTS.md L2 / public artifact hygiene Error string is bounded and credential-safe; no API keys or raw env data Return generic synthesis_failed reason if message is unsafe PR body documents bounded behavior Unit test asserts no secret-shaped payload is required

Decision Record impact

none. This is a service-level degradation-path bug fix aligned with the existing KB/MX direction; it does not amend an ADR.

Acceptance Criteria

  • SearchService.ask returns ranked references when model.generateContent() rejects after retrieval succeeds.
  • The returned answer explicitly marks synthesis as unavailable/degraded so the caller does not mistake it for a normal synthesized answer.
  • The response includes a machine-readable degraded marker.
  • Retrieval/no-result behavior remains unchanged.
  • Focused unit coverage exists for the synthesis-failure path without using live Gemini.
  • Focused unit coverage for existing SearchService hydration behavior still passes.

Out of Scope

  • Reworking QueryService ranking or candidate weighting; that is covered by #12719.
  • Lexical-rescue index performance; that is covered by #12715.
  • Root-causing Chroma Error finding id / query_summaries corruption; that is covered by #12450.
  • Switching generation providers or changing quota policy.

Avoided Traps

  • Retrying Gemini inside ask: rejected for this ticket. Retry/backoff can amplify quota exhaustion and still leaves callers blind if retries fail.
  • Telling agents to use query_documents instead: rejected. ask_knowledge_base is intentionally the primary tool; the tool should degrade well instead of pushing every caller into manual fallback.
  • Catching all errors around retrieval + synthesis together: rejected. Retrieval failures are materially different from synthesis failures and must remain visible.

Related

Related: #12696

Origin Session ID: e8f07ef9-ef7e-4815-8ff4-7abe13720621 Retrieval Hint: "ask_knowledge_base synthesis provider quota fallback SearchService references degraded"

tobiu closed this issue on Jun 8, 2026, 10:52 AM
tobiu referenced in commit 3204a8c - "fix(knowledge-base): degrade ask synthesis failures (#12737) (#12738)" on Jun 8, 2026, 10:52 AM