LearnNewsExamplesServices
Frontmatter
id8342
titleSwitch Knowledge Base to Source Code Indexing (SourceParser)
stateClosed
labels
enhancementai
assigneestobiu
createdAtJan 5, 2026, 11:57 PM
updatedAtJan 6, 2026, 12:04 AM
githubUrlhttps://github.com/neomjs/neo/issues/8342
authortobiu
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJan 6, 2026, 12:04 AM

Switch Knowledge Base to Source Code Indexing (SourceParser)

Closed v11.19.0 enhancementai
tobiu
tobiu commented on Jan 5, 2026, 11:57 PM

Switch the API knowledge source from docs/output/all.json (JSDoc JSON) to raw src/**/*.mjs files.

Goal: Provide the AI with implementation details and context, not just API signatures. Reduce vector cardinality by grouping cohesive units.

Plan:

  1. Create ai/mcp/server/knowledge-base/parser/SourceParser.mjs:

    • Use acorn to parse .mjs files.
    • Chunking Strategy:
      • Module Context: Imports + Module-level variables (e.g., const symbol = ...) + Class Header (JSDoc + Class definition).
      • Class Properties: Static and instance fields defined on the class body (e.g., static observable = true, myProp = 0) excluding the config object.
      • Config Block: The entire static config = { ... } object as a single chunk.
      • Methods: Individual methods (Body + JSDoc).
    • Metadata: Include line numbers (line_start, line_end) for future "open in editor" features.
  2. Update ApiSource.mjs:

    • Change target from docs/output/all.json to src/.
    • Use SourceParser for parsing .mjs files.
    • Retain deterministic sorting.
  3. Cleanup:

    • Remove ApiParser.mjs (the JSDoc JSON parser).

Note: This relies on the acorn dependency which is already present (used by TestParser).

tobiu added the enhancement label on Jan 5, 2026, 11:57 PM
tobiu added the ai label on Jan 5, 2026, 11:57 PM
tobiu assigned to @tobiu on Jan 5, 2026, 11:57 PM
tobiu referenced in commit 0e0ac53 - "Enhancement: Switch Knowledge Base to Source Code Indexing #8342 on Jan 6, 2026, 12:03 AM
tobiu
tobiu Jan 6, 2026, 12:03 AM

Input from Gemini 3 Pro:

✦ I have successfully switched the Knowledge Base indexing strategy from JSDoc JSON to raw Source Code.

Changes Implemented:

  1. Created SourceParser.mjs:

    • Uses acorn to parse .mjs files into an AST.
    • Chunking Strategy:
      • Module Context: Captures imports, top-level variables, and the class header (JSDoc + Class Declaration).
      • Class Properties: Groups static and instance fields (e.g., static observable, myProp = 0) that are not part of the config block.
      • Config Block: Extracts the entire static config = { ... } object as a single, cohesive chunk.
      • Methods: Extracts individual methods (Body + JSDoc) with line number metadata.
  2. Updated ApiSource.mjs:

    • Now targets the src/ directory instead of docs/output/all.json.
    • Recursively scans for .mjs files.
    • Uses SourceParser to generate chunks.
    • Maintains the previously added deterministic sorting.
  3. Deleted ApiParser.mjs:

    • Removed the obsolete parser that relied on jsdoc-api output.

This change upgrades the AI's understanding from purely API-based (signatures and descriptions) to implementation-based (logic and patterns), while also reducing vector cardinality by grouping related configurations.

tobiu closed this issue on Jan 6, 2026, 12:04 AM
tobiu cross-referenced by #8420 on Jan 8, 2026, 1:02 PM