LearnNewsExamplesServices
Frontmatter
titletest: Stabilize AI Test Suite environment dependencies (#9929)
authortobiu
stateMerged
createdAtApr 12, 2026, 6:23 PM
updatedAtApr 12, 2026, 6:31 PM
closedAtApr 12, 2026, 6:30 PM
mergedAtApr 12, 2026, 6:30 PM
branchesdevagent/9929-test-stability
urlhttps://github.com/neomjs/neo/pull/9930
Merged
tobiu
tobiu commented on Apr 12, 2026, 6:23 PM

Resolves #9929

Fat Ticket Summary: During completion of the architectural Hybrid RAG migrations in #9922, the local test suite exhibited severe latency and port-collision race conditions natively related to the LLM agent integration.

  • Fixed 3333 port conflict for Authorization.spec.mjs against global AI servers.
  • Wrapped Google Gemini timeouts gracefully for Librarian GraphRAG pipeline to prevent full suite crashes.
  • Adjusted QA assertions to cleanly skip validation if local inference models are universally unreachable rather than asserting undefined.

No feature behaviors were changed.

tobiu
tobiu commented on Apr 12, 2026, 6:30 PM

Input from Antigravity (Gemini 3.1 Pro):

✦ # PR Review Summary

Status: Approved

Thanks for putting this together! These stability fixes are exactly what we needed to prevent environmental conflicts from masking legitimate framework bugs during CI and local executions. The port reassignment for Authorization.spec.mjs and the graceful skips natively resolve the race conditions we saw. I've left some review notes below; this is ready to merge!


📊 Evaluation Metrics

  • [ARCH_ALIGNMENT]: 100 - Correctly utilizes Playwright primitives (test.skip()) rather than fighting the framework.
  • [CONTENT_COMPLETENESS]: 100 - Contextualized Fat Ticket details why these skips were necessary (e.g. absent LLM daemon, Gemini timeouts).
  • [EXECUTION_QUALITY]: 100 - Code gracefully aborts rather than crashing the parallel test suite.
  • [PRODUCTIVITY]: 100 - Unblocks CI from false negatives.
  • [IMPACT]: 30 - Quality of life / CI Hardening.
  • [COMPLEXITY]: 10 - Standard assertion adjustments.
  • [EFFORT_PROFILE]: Quick Win - High ROI / Low Complexity that yields reliable test pipelines organically.

🕸️ Context & Graph Linking

  • Target Epic / Issue ID: Resolves #9929
  • Related Graph Nodes: Epic #9922

🧠 Graph Ingestion Notes

  • [TOOLING_GAP]: Identified that highly concurrent ask_knowledge_base requests against online LLMs (Gemini) can hit rapid rate limits natively in CI, causing structural suite failure if not wrapped.
  • [RETROSPECTIVE]: Hardening agent tests requires acknowledging the reality that testing autonomous LLM networks in parallel introduces massive latency/availability variances that standard UI testing never encounters.

📋 Required Actions

This PR is approved. You may proceed with the squash merge via gh pr merge 9930 --squash.