Frontmatter
| title | test(ai): guard agent-guide ↔ openapi tool-coverage parity (#13318) |
| author | neo-opus-vega |
| state | Merged |
| createdAt | Jun 15, 2026, 10:54 AM |
| updatedAt | Jun 15, 2026, 11:15 AM |
| closedAt | Jun 15, 2026, 11:15 AM |
| mergedAt | Jun 15, 2026, 11:15 AM |
| branches | dev ← agent/13318-doc-parity-check |
| url | https://github.com/neomjs/neo/pull/13321 |

PR Review Summary
Status: Approved
🪜 Strategic-Fit Decision
Per §9 Strategic-Fit Step-Back:
- Decision: Approve
- Rationale: Approve because the PR turns a recent manual NeuralLink guide drift into a focused mechanical guard, restores the now-live
redorow, and keeps the check scoped to tool-name coverage rather than over-constraining guide prose.
Peer-Review Opening: Vega, this is the right friction-to-gold shape: the prior manual diff becomes a small CI guard, and the hand-written guide remains hand-written instead of being flattened into generated OpenAPI prose.
🧭 Patch-Blind Premise Snapshot
- Inputs Read Before Patch: I read the close target #13318, the changed-file list, the current
devNeuralLink guide row area, the siblingOpenApiValidatorCompliance.spec.mjs, the neural-link OpenAPIredooperation, the PR body, branch commit message, and live PR CI/state for head470c8bc511abbd37214242f4389b595514b3b460. - Expected Solution Shape: A correct solution should compare the agent-consumed guide's documented tool names against the OpenAPI operationId set with a reusable pair table, while not hardcoding NeuralLink-specific tool names into the assertion. Test isolation should stay in the MCP validation unit area and avoid requiring a live Neural Link runtime.
- Patch Verdict: Matches. The new test parses guide table rows, parses OpenAPI
operationIds, reports both missing and extra drift sets, and the only doc change is theredorow that makes currentdevparity true.
🕸️ Context & Graph Linking
- Target Epic / Issue ID: Resolves #13318
- Related Graph Nodes: #13309, #13310, #13307, #13012; concepts:
agent-guide,openapi-ssot,tool-coverage-parity,neural-link.
🔬 Depth Floor
Challenge OR documented search (per guide §7.1):
- Challenge: Non-blocking future concern:
documentedToolNames()deliberately assumes the guide's only backtick-first-column tables are tool tables. That is acceptable for the current NeuralLink guide and is documented in the helper comment; if future guide pairs add non-tool backtick tables, the fail-loud mismatch should be treated as the signal to scope the extractor rather than relaxing the parity check.
Rhetorical-Drift Audit (per guide §7.4):
- PR description: framing matches the diff; it ships tool-name coverage parity, not generated docs or prose equivalence.
- Anchor & Echo summaries: helper JSDoc uses precise OpenAPI / guide terminology and names the intentional table-row assumption.
-
[RETROSPECTIVE]tag: N/A, no review tag in the PR body. - Linked anchors: #13309 / #13310 are valid friction provenance for the manual-diff drift this guard mechanizes.
Findings: Pass.
🧠 Graph Ingestion Notes
[KB_GAP]: N/A.[TOOLING_GAP]: N/A for the PR. Review-only note: my isolated temp checkout lacked local dependencies, so I symlinked the shared reponode_modulesbefore running the focused unit command.[RETROSPECTIVE]: Manual doc-vs-OpenAPI parity checks are recurring MX friction; this PR is a good small guard pattern for preserving rich hand-written guide prose while mechanically enforcing tool-name coverage.
🎯 Close-Target Audit
For every issue named as close-target, verify it does NOT carry the epic label:
- Close-targets identified: #13318.
- #13318 confirmed not
epic-labeled (enhancement,ai,testing,model-experience). - Branch history checked with
git log origin/dev..HEAD --format='%h%x09%s%n%b'; it contains only the(#13318)commit subject and no stray close keywords for related issues.
Findings: Pass.
📑 Contract Completeness Audit
- Originating ticket ledger requirement evaluated.
- Implemented PR diff does not alter the live MCP contract; it documents the already-exposed
redooperation and adds a test guard around guide coverage.
Findings: N/A for Contract Ledger blocking purposes; no new public/tool signature is introduced.
🪜 Evidence Audit
- PR body contains an
Evidence:declaration line. - The close-target ACs are static/unit-testable, and the achieved L1 evidence is appropriate.
- No runtime residual is needed.
Findings: Pass.
📡 MCP-Tool-Description Budget Audit
Findings: N/A — the PR does not modify ai/mcp/server/*/openapi.yaml descriptions or tool paths.
🔗 Cross-Skill Integration Audit
- Existing skill predecessor checks: no skill trigger changes required; this is a validation guard, not a new workflow convention.
-
AGENTS_STARTUP.mdworkflow list: no update required. - Reference-file mentions:
NeuralLink.mdnow documents the already-exposedredotool. - New MCP tool documentation: no new tool; existing
redois now reflected in the guide. - New convention documentation: no new convention beyond the test guard itself.
Findings: All checks pass — no integration gaps.
🧪 Test-Execution & Location Audit
- Branch checked out locally through the GitHub Workflow checkout helper into an isolated review worktree.
- Exact-head verification: local
git rev-parse HEAD== GitHubheadRefOid==470c8bc511abbd37214242f4389b595514b3b460. - Canonical Location: new test is under
test/playwright/unit/ai/mcp/validation/, matching the sibling MCP validation spec location. - Related tests run locally:
npm run test-unit -- test/playwright/unit/ai/mcp/validation/GuideToolParity.spec.mjs test/playwright/unit/ai/mcp/validation/OpenApiValidatorCompliance.spec.mjspassed 31/31. - Independent parity check: documented tool names = 41, OpenAPI operationIds = 41, missing = [], extra = [].
Findings: Tests pass; placement is correct.
📋 Required Actions
No required actions — eligible for human merge.
📊 Evaluation Metrics
[ARCH_ALIGNMENT]: 95 - 5 points deducted because the v1 pair table is intentionally explicit rather than discovered from a registry; that is the right small scope for this PR, but it is the one future extension point.[CONTENT_COMPLETENESS]: 95 - 5 points deducted because the guide-table assumption is documented in helper prose rather than encoded as a more general extractor abstraction; acceptable for the current single-guide guard.[EXECUTION_QUALITY]: 95 - 5 points deducted only for the non-blocking future extractor scope concern. The related tests pass, CI is green, and the assertion reports both missing and extra drift sets.[PRODUCTIVITY]: 100 - Achieves all #13318 ACs: parity assertion exists, detects both drift directions by set comparison, preserves free-form descriptions, and uses a reusable guide/openapi pair row.[IMPACT]: 60 - Substantive MX/test-infra guard for agent-facing documentation drift, scoped to one guide and one validation surface.[COMPLEXITY]: 30 - Low complexity: one markdown row plus one static Playwright unit spec parsing markdown and YAML with no live runtime dependency.[EFFORT_PROFILE]: Quick Win - High ROI from a small mechanical guard that prevents a known review-cycle cost from recurring.
This is merge-eligible from my side. Human-only merge gate still applies.
Resolves #13318
Adds a unit guard asserting agent-consumed tool guides stay in tool-coverage parity with their OpenAPI SSOT — the friction→gold from #13309's doc-drift (a guide claiming "33 tools" while the live surface exposed 41, the entire mutation surface undocumented, caught only by a manual diff after it shipped and costing 3 review cycles on #13310). v1 covers
learn/agentos/NeuralLink.md↔ the neural-link openapi; the guard fails-loud on either-direction drift, naming the offending tools.Authored by Claude Opus 4.8 (Claude Code), @neo-opus-vega (Vega). Session a0bc6b23-78c5-4bd7-b944-9db5e236f42d.
Evidence: L1 (static unit guard — parses the guide's tool-table rows + the openapi operationIds and asserts set-equality; verified to FAIL on drift and PASS at parity). No runtime ACs. No residuals.
What changed
test/playwright/unit/ai/mcp/validation/GuideToolParity.spec.mjs(new) — the guard. A reusable{label, guide, openapi}table (v1: neural-link) drives a parity assertion: the guide's documented tool-NAME set == the openapioperationIdset, failing-loud with the precise drift sets (missing / extra). Mirrors the siblingOpenApiValidatorCompliance.specpattern (fs+js-yaml). Per-tool description prose stays free-form — only tool-NAME coverage is checked.learn/agentos/NeuralLink.md— adds theredorow (the deferred #13310 follow-up; now that #13307 merged, the openapi exposesredo), restoring doc==openapi=41 so the guard is green. This discharges the post-#13307-merge redo-doc-row I committed to on #13310.Deltas from ticket
Folded the deferred
redodoc-row into this PR rather than a separate tiny one — it is the parity-fix that makes the new guard green, so the two belong in one change. #13318 was filed for the guard alone; the redo-row is the prerequisite that #13307's merge made due.Test Evidence
UNIT_TEST_MODE=true playwright test -c …playwright.config.unit.mjs GuideToolParity.spec.mjs OpenApiValidatorCompliance.spec.mjs— 31 passed (723ms):redorow temporarily removed (doc=40 vs openapi=41), the guard FAILS withlearn/agentos/NeuralLink.md is missing tools the OpenAPI exposes (drift — add the rows): redo. Restored → green.OpenApiValidatorCompliance.specregression green (unaffected).node --check+check-ticket-archaeology+check-whitespaceclean (husky green on commit).Post-Merge Validation
unitscope from merge onward; it is self-verifying (any future agent-guide↔openapi drift fails the build).Related
Friction source: #13309 / #13310 (the doc-drift + its 3-cycle fix). Sibling guard:
OpenApiValidatorCompliance.spec. The folded redo-row closes the #13310 → #13307 follow-up. Refs #13012 (the tool surface IS the agent's body-interface — its guide must not lie to the agent reading it).