LearnNewsExamplesServices
Frontmatter
titledocs: Promote ask_knowledge_base as primary Anti-Hallucination tool (#9893)
authortobiu
stateMerged
createdAtApr 11, 2026, 9:56 PM
updatedAtApr 11, 2026, 10:04 PM
closedAtApr 11, 2026, 10:04 PM
mergedAtApr 11, 2026, 10:04 PM
branchesdevagent/9893-promote-ask-knowledge-base
urlhttps://github.com/neomjs/neo/pull/9894
Merged
tobiu
tobiu commented on Apr 11, 2026, 9:56 PM

Summary

Resolves #9893 Partially addresses #9892 (CodebaseOverview scope extension per comment)

Changes

AGENTS.md — §2 Anti-Hallucination Policy (Major)

  • Expanded from 7 lines to a structured subsection with §2.1 Tool Hierarchy table positioning ask_knowledge_base as the #1 tool
  • Added §2.2 Anti-Patterns with concrete bad/good examples including the query_documents → read 5 files waste pattern

openapi.yaml — ask_knowledge_base description (Major)

  • Expanded from 12 lines to ~50 lines matching query_documents documentation quality
  • Added: Priority Positioning, How It Works (zero-cost RAG subagent framing), Example Queries, decision table (when to use this vs. query_documents), Anti-Pattern Warning

AGENTS_STARTUP.md — §3.4 Ask the Expert Protocol (Minor)

  • Cross-references AGENTS.md §2.1 tool hierarchy
  • Adds session initialization framing (use during warm-up)
  • Added third example query

CodebaseOverview.md — Query Entry Points + When Stuck (Minor, from #9892 comment)

  • "Query Entry Points" intro now leads with "Ask, Don't Search" callout for ask_knowledge_base
  • "When Stuck" section reordered: ask first → then locate files

Rationale

ask_knowledge_base is architecturally the most powerful anti-hallucination primitive — a zero-cost RAG subagent that even local SLMs can invoke. Despite this, it had zero mentions in AGENTS.md, a 12-line OpenAPI description (vs 70+ for query_documents), and no mention in CodebaseOverview. Agents systematically defaulted to the multi-file-read pattern, wasting context window.

Scope Deviation

Added CodebaseOverview.md changes (from #9892 comment scope extension) that were not in the original #9893 body. These are logically inseparable — promoting the tool in agent protocols without updating the primary onboarding document would leave a gap.

tobiu
tobiu commented on Apr 11, 2026, 10:00 PM

Input from Claude Opus 4.6 (Antigravity):

Iterative Update (force-push):

Fixed CodebaseOverview.md "When Stuck" step 2 per user review. The previous wording suggested using query_documents as a follow-up to locate files, but ask_knowledge_base already returns ranked file references with full paths and scores in its response. The redundant query_documents call was replaced with grep_search / view_file as the natural next step for drilling into specific implementations.

-1. **Ask first:** ... — gets a synthesized answer with citations
-2. **Then locate files:** `query_documents(query='afterSet hook', type='src')` — if you need the actual source code
+1. **Ask first:** ... — gets a synthesized answer with cited source file paths
+2. **Drill into code:** The answer includes ranked file references — use `grep_search` or `view_file` to inspect the specific implementation

tobiu
tobiu commented on Apr 11, 2026, 10:01 PM

Input from Claude Opus 4.6 (Antigravity):

❋ # PR Review Summary

Status: Approved

Clean documentation-only PR. Four files surgically edited to promote ask_knowledge_base from an undocumented tool to the #1 position in the anti-hallucination hierarchy. The PR body is a proper Fat Ticket with rationale, scope deviation documentation, and cross-references. Iterative force-push was correctly documented via comment.


📊 Evaluation Metrics

  • [ARCH_ALIGNMENT]: 95 - Correctly positions ask_knowledge_base as a zero-cost RAG subagent within the existing Anti-Hallucination framework. The tool hierarchy table in AGENTS.md §2.1 follows the established cascading-priority pattern used across other protocol sections. Minor deduction: the openapi.yaml decision table uses ❌/✅ which may not render in all YAML consumers.
  • [CONTENT_COMPLETENESS]: 90 - All four documentation surfaces are covered (AGENTS.md, AGENTS_STARTUP.md, openapi.yaml, CodebaseOverview.md). Cross-references between files are bidirectional. The iterative PR comment documents the force-push delta. Deduction: no AGENTS.md §6 (Request Triage) update — that section says "using the knowledge base and other tools" generically and could also benefit from the ask_knowledge_base callout.
  • [EXECUTION_QUALITY]: 95 - YAML parse validated via node. No formatting regressions. The "When Stuck" fix (removing redundant query_documents step) was caught by user review and correctly addressed. All edits are additive — no existing semantics were broken.
  • [PRODUCTIVITY]: 100 - All goals of #9893 achieved plus the #9892 comment scope extension. The user's review feedback was incorporated in-session.
  • [IMPACT]: 75 - This is a force-multiplier for all future agent sessions. Every agent that reads AGENTS.md (i.e., every agent) will now default to ask_knowledge_base instead of the wasteful query_documents → read 5 files chain. However, the actual behavioral change depends on whether LLM providers weight OpenAPI descriptions and protocol docs equally in tool selection.
  • [COMPLEXITY]: 20 - Documentation-only. Four files, 74 insertions, 12 deletions. No code logic, no tests, no architectural risk.
  • [EFFORT_PROFILE]: Quick Win - High ROI (affects every agent session) with minimal complexity and zero regression risk.

🕸️ Context & Graph Linking

  • Target Epic / Issue ID: Resolves #9893
  • Related Graph Nodes: #9892 (CodebaseOverview expansion — partially addressed via the Query Entry Points / When Stuck changes), #9891 (Strategic Constraint Nodes — downstream consumer of better tool selection), ask_knowledge_base operationId, query_documents operationId

🧠 Graph Ingestion Notes

  • [KB_GAP]: The ask_knowledge_base tool existed since the KB MCP server's inception but had zero references in agent operational protocols (AGENTS.md). This gap caused systematic under-utilization — agents defaulted to query_documents because it had 70+ lines of OpenAPI documentation vs 12 for ask_knowledge_base. Root cause: tool documentation asymmetry directly drives tool selection in LLMs.
  • [TOOLING_GAP]: get_pull_request_diff MCP tool failed with exit code 1 for PR #9894 during this review. Fell back to gh pr diff via shell. Worth investigating — may be a timing issue with recently force-pushed PRs.
  • [RETROSPECTIVE]: This PR validates a meta-architectural insight: documentation IS architecture for AI-native systems. The behavioral change isn't in code — it's in how future agents will select tools based on the weight of documentation surrounding each tool. The "zero-cost RAG subagent" framing is a conceptual anchor that should be reinforced in future onboarding materials.

📋 Required Actions

None — approved for merge.

Optional enhancement for a follow-up commit: Consider adding an ask_knowledge_base callout to AGENTS.md §6 (Request Triage), which currently says "using the knowledge base and other tools" generically. This would close the last documentation surface that references KB tools without specifying the priority order.