LearnNewsExamplesServices
Frontmatter
id11687
titleAdd cross-server MCP boot and listTools smoke coverage
stateClosed
labels
enhancementaitestingmodel-experience
assigneesneo-gpt
createdAtMay 20, 2026, 5:31 PM
updatedAtMay 20, 2026, 9:48 PM
githubUrlhttps://github.com/neomjs/neo/issues/11687
authorneo-gpt
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMay 20, 2026, 9:48 PM

Add cross-server MCP boot and listTools smoke coverage

Closedenhancementaitestingmodel-experience
neo-gpt
neo-gpt commented on May 20, 2026, 5:31 PM

Context

This ticket converts a concrete review-loop friction point into CI coverage. During the #11680 / #11681 get_neighbors tool-shape follow-up, the operator called out the failure mode directly: if a change affects an MCP tool shape, the matching openapi.yaml must move with it or the MCP server may fail at boot/list-tools time. The requested shape is explicitly separate from the active implementation PR: add a new ticket for low-cognitive-load tests that boot/list Neo MCP servers and make CI go red when the tool surface is structurally broken.

The Problem

Agents and reviewers do not have a cheap, uniform answer to: "did this change break one of the Neo MCP server tool surfaces?" We can inspect diffs and individual service tests, but MCP exposure is assembled through per-server toolService.mjs + openapi.yaml wiring. A missing YAML field, invalid schema, stale path, or broken service mapping can survive local service-level tests and only surface when a harness asks the server for its tools.

Duplicate sweep result: #11110 is the closest prior artifact, but it is scoped to github-workflow after the M6 migration and explicitly leaves other MCP servers out of scope. This ticket is the broader cross-server smoke layer.

The Architectural Reality

V-B-A evidence gathered before filing:

  • Current MCP server entrypoints/specs exist for five servers: ai/mcp/server/neural-link, ai/mcp/server/knowledge-base, ai/mcp/server/github-workflow, ai/mcp/server/file-system, and ai/mcp/server/memory-core; each has mcp-server.mjs and openapi.yaml.
  • ai/mcp/server/BaseServer.mjs wires MCP tools/list to toolService.listTools({cursor, limit}) and maps each returned tool's name, inputSchema, outputSchema, and annotations into the MCP response.
  • Existing coverage is asymmetric: test/playwright/unit/ai/mcp/server/memory-core/McpServerToolLimits.spec.mjs calls listTools() for Memory Core, while test/playwright/unit/ai/mcp/server/github-workflow/ToolRegistration.spec.mjs performs static serviceMapping regex checks for a few tool names. There is no shared all-server listTools smoke harness.
  • #11110 documents the same bug class for github-workflow: boot-only checks can miss lazy OpenAPI/listTools failures; per-service tests do not exercise MCP wiring.

The Fix

Add a shared MCP smoke-test layer under test/playwright/unit/ai/mcp/server/ that enumerates the active Neo MCP servers and exercises their tool-listing surface.

Recommended implementation shape:

  1. Add a reusable server fixture list for active MCP servers, including server name, mcp-server.mjs, toolService.mjs, and expected openapi.yaml path.
  2. Add a fast unit smoke that imports each server's toolService.mjs, calls listTools(), and asserts:
    • no import/listTools error,
    • tool count is non-zero,
    • every tool has a valid name and inputSchema,
    • declared tool names are unique per server,
    • the generated tool list is compatible with the server's openapi.yaml operation IDs.
  3. Where boot can be isolated without external daemons, add a stdio MCP handshake smoke per server that starts mcp-server.mjs, sends initialize, then tools/list, and fails cleanly on boot crash or malformed list response.
  4. If a server cannot participate in stdio boot smoke without heavy dependencies, document the reason in the fixture and keep the fast toolService.listTools() guard mandatory.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
MCP tools/list output for every active Neo MCP server ai/mcp/server/*/openapi.yaml + per-server toolService.mjs Unit CI exercises all active server tool-listing surfaces and fails on YAML parse, stale path, duplicate name, or mapping drift Server-specific fixture note only when stdio boot requires unavailable external dependencies Test names and comments should point to #11110 and this ticket npm run test-unit -- test/playwright/unit/ai/mcp/server

Acceptance Criteria

  • A shared MCP server fixture enumerates all currently active Neo MCP servers: memory-core, knowledge-base, github-workflow, neural-link, and file-system.
  • Unit coverage calls listTools() for every fixture and asserts non-empty, well-formed, uniquely named tools.
  • Unit coverage verifies each server's listed tool names align with its openapi.yaml operation IDs, so YAML/toolService drift fails in CI.
  • At least one reusable stdio tools/list handshake helper exists; each server either uses it or documents a specific dependency reason why only the fast toolService smoke applies.
  • The new tests run through npm run test-unit -- test/playwright/unit/ai/mcp/server without requiring live Chroma, GitHub credentials, browser windows, or harness-local MCP config.
  • Existing targeted tests such as Memory Core tool-limit coverage and GitHub Workflow registration checks are either reused or left intact; this ticket does not delete narrower regression tests.

Out of Scope

  • Fixing any discovered MCP schema or server boot failure beyond the first small unblock needed to make the new smoke tests meaningful.
  • Replacing #11110's deeper github-workflow boot/wiring/integration coverage.
  • Changing MCP tool schemas, openapi.yaml descriptions, or service mappings except as required by a failing smoke test discovered during implementation.
  • Adding new MCP tools.
  • Running dockerized integration stacks as part of the fast unit smoke layer.

Avoided Traps / Gold Standards Rejected

  • Only test the recently touched tool. Rejected because the user-facing friction is cognitive load: reviewers do not always know which service methods become MCP tools. The listTools surface already knows.
  • Only boot the process. Rejected because #11110 documents that boot-only checks can miss lazy OpenAPI/listTools failures.
  • Full integration-only coverage. Rejected because the goal is a cheap CI warning. Dockerized integration remains valuable, but this guard should fail fast in unit CI.
  • One-off regex checks per server. Rejected because they reproduce the current asymmetry. A shared fixture prevents future servers from skipping the guard by accident.

Related

  • #11110 — github-workflow MCP needs systematic boot + wiring + integration test coverage
  • #11680getNeighbors output schema follow-up that exposed the latest YAML/tool-shape review friction
  • PR #11681 — immediate review cycle that triggered the operator clarification
  • AGENTS.md §friction_to_gold and §verify_before_assert

Origin Session ID: 019e44ba-d309-7e91-a819-36911fbf4e10 Retrieval Hint: "cross-server MCP listTools smoke", "boot each Neo MCP server", "openapi yaml toolService drift", "MCP tools/list CI guard"

tobiu referenced in commit c9be033 - "feat(mcp): add cross-server listTools smoke coverage (#11687) (#11689) on May 20, 2026, 9:48 PM
tobiu closed this issue on May 20, 2026, 9:48 PM