[enhancement] ask_knowledge_base query telemetry — Agent FAQ signal for gap inference

The Problem / Why

Neo.mjs's Agent OS currently captures two agent observation axes:

1. Trajectories — session memory (ChromaDB, via add_memory per turn)
2. Observations — PR review tags ([KB_GAP], [TOOLING_GAP], [RETROSPECTIVE]) ingested by DreamService as graph edges

A third signal exists but is discarded: agent questions. Every ask_knowledge_base call is an agent saying "I don't know this thing and am asking the KB" — a direct, unfiltered, high-signal expression of knowledge gaps. Currently these queries are stateless; nothing accumulates.

High-frequency unresolved queries are the single most direct evidence that documentation is thin or missing. Stack Overflow's most-asked questions drive prioritization; Google Analytics's empty-search-result queries drive content creation. Applied to Neo.mjs's closed agent loop, this becomes a self-correcting documentation-demand signal.

Canonical Precedent: Neural Link RecorderService

This is not a new pattern to invent — it is an existing pattern to replicate.

The Neural Link server already implements the exact shape this ticket describes, via ai/mcp/server/neural-link/services/RecorderService.mjs + its call-site wrapper at ai/mcp/server/neural-link/services/toolService.mjs (lines 62–104). Every NL tool invocation is captured to a SQLite nl_action_log table (schema: id, agent_id, session_id, sequence_id, timestamp, tool, args, result, success, duration_ms, app_name, reward) via a finally block that runs regardless of success or error. WAL journal mode, indexed on sequence_id, session_id, timestamp.

This ticket is "do the same thing at the knowledge-base server."

Proposed Architecture

Instrumentation Site (corrected)

Per-server callTool wrapper in ai/mcp/server/knowledge-base/services/toolService.mjs, mirroring the exact pattern from ai/mcp/server/neural-link/services/toolService.mjs:62-104:

const _callTool = toolService.callTool.bind(toolService);

const callTool = async (name, args) => {
    const t0 = Date.now();
    let result, success = 0;
    try {
        result  = await _callTool(name, args);
        success = 1;
        return result;
    } catch (err) {
        result = { error: err.message };
        throw err;
    } finally {
        KBRecorderService.log({
            timestamp: t0, tool: name, args: safeStringify(args),
            result: safeStringify(result), success,
            duration_ms: Date.now() - t0,
            agent_id: /* from context */, session_id: /* from context */
        });
    }
};

NOT at ai/mcp/ToolService.mjs — that's the shared Zod/OpenAPI validation layer, not the instrumentation layer. The original framing in this ticket's first revision was wrong; corrected after reading the precedent.

Storage (corrected)

SQLite, not JSONL. Either:

• Option A (preferred): new table kb_query_log in the shared Memory Core SQLite DB (same file RecorderService already writes to). Schema parallels nl_action_log plus a query_text column (denormalized from args for faster read-path) and eventually an embedding_hash column for dedup indexing.
• Option B: reuse nl_action_log directly, filter by tool IN ('ask_knowledge_base', 'query_documents', …) at read time. Simpler but couples two subsystems.

Lean A — cleaner separation of concerns, room to grow FAQ-specific columns (embedding, cluster_id) without polluting the NL action schema.

Deduplication / FAQ Aggregation

Background clustering via cosine similarity, using existing KB ChromaDB embedding pipeline (zero new embedding infra). Start at ~0.85 threshold as a calibration point — measure and adjust empirically, don't hardcode (per feedback_challenge_numerical_thresholds_in_acs.md).

Output: new table kb_query_faqs in the same SQLite DB:

• cluster_id (primary key)
• canonical_query (representative text)
• variants (JSON array of query texts)
• count (occurrence count)
• first_seen, last_seen (timestamps)
• related_concept_ids (JSON array, joined via ConceptService's classifyConcept against query text)
• has_strong_guide_coverage (boolean, computed from concept graph)

Read Path

New MCP tool on neo-mjs-knowledge-base: list_agent_faqs({limit, minCount}) — returns top FAQs by count. Observability for humans + agents; not action-taking.

Gap Inference Integration

DreamService's GapInferenceEngine consumes the kb_query_faqs table:

• If a cluster has count >= threshold AND has_strong_guide_coverage = false → emit a new gap category [KB_DEMAND_GAP] (distinct from structural [GUIDE_GAP])
• Feeds into Golden Path synthesis alongside existing gap signals

Explicit Scope Exclusions

• No auto-generation of guide content from FAQs. Ship the observability + gap signal first. Auto-generation is a separate ticket contingent on FAQ signal proving reliable at ≥3 months of data.
• No retroactive mining of past memories for historical queries. Start collecting from deployment forward.
• Not universal to all MCP servers in Phase 1. Knowledge-base server gets the Recorder first (FAQ is the immediate value). If the pattern proves useful, extending to memory-core / github-workflow is a follow-up ticket.

Acceptance Criteria

• KBRecorderService added at ai/mcp/server/knowledge-base/services/KBRecorderService.mjs, mirroring the NL RecorderService shape
• Per-server callTool wrapper in ai/mcp/server/knowledge-base/services/toolService.mjs (or equivalent) calls KBRecorderService.log({...}) in its finally block
• SQLite kb_query_log table with WAL journal + indexes matching RecorderService pattern
• Background clustering daemon (or on-demand CLI via buildScripts/ai/) produces kb_query_faqs table entries
• New MCP tool list_agent_faqs exposed on the knowledge-base server
• Cluster similarity threshold documented as calibration starting point with measurement plan
• Playwright tests covering: telemetry capture on tool call, cluster dedup, FAQ aggregation, MCP tool response
• GapInferenceEngine integration emits [KB_DEMAND_GAP] tag when FAQs lack EXPLAINED_BY

Related

• #10030 — parent epic (fed into by the new gap signal)
• ai/mcp/server/neural-link/services/RecorderService.mjs — canonical precedent (pattern to replicate)
• ai/mcp/server/neural-link/services/toolService.mjs:62-104 — canonical call-site wrapper pattern
• DreamService.mjs / GapInferenceEngine — downstream consumer
• feedback_challenge_numerical_thresholds_in_acs.md — discipline applied to similarity threshold
• feedback_verify_written_claims_against_precedent.md — discipline that produced this correction (the first revision of this ticket cited ai/mcp/ToolService.mjs from docs without checking the NL precedent)

Origin Session ID

62d6f155-e57f-4279-9b59-36c9e4ecbc5e

Revision Note

This ticket body was rewritten on 2026-04-19 after tobi's challenge on the instrumentation site. The first revision proposed the shared ai/mcp/ToolService.mjs and JSONL storage; both were wrong. RecorderService exists; this ticket is "do the same thing at the KB server."