LearnNewsExamplesServices
Frontmatter
id15576
titleDecouple pure unit tests from Chroma startup
stateClosed
labels
bugwindowscontributor-experienceaitestingarchitecture
assigneesneo-gpt
createdAtJul 19, 2026, 5:04 PM
updatedAtJul 19, 2026, 9:38 PM
githubUrlhttps://github.com/neomjs/neo/issues/15576
authorneo-gpt
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 19, 2026, 9:38 PM

Decouple pure unit tests from Chroma startup

Closed Backlog/active-chunk-8 bugwindowscontributor-experienceaitestingarchitecture
neo-gpt
neo-gpt commented on Jul 19, 2026, 5:04 PM

Context

A community contributor claimed #15429 and immediately hit a native Windows x64 startup failure while trying to run the ticket's canonical pure-utility command:

npm run test-unit -- test/playwright/unit/util/Matrix.spec.mjs

This is not a Matrix-test defect. It exposes a test-harness boundary leak: every unit-test invocation currently boots the Agent OS vector-store daemon, even when the selected spec exercises only src/util/**.

The Problem

test/playwright/playwright.config.unit.mjs:14-24 allocates Chroma host, port, and data-dir state for every run, and lines 35-46 make chroma run an unconditional Playwright webServer. The contributor therefore cannot reach a pure Body test on native Windows x64.

The reported upstream premise is verified, not inferred:

  • Neo currently depends on chromadb@3.5.0.
  • The installed JS CLI rejects win32/x64 before startup even though the package metadata includes a Windows x64 optional binding.
  • The upstream bug remains open as chroma-core/chroma#5188.
  • learn/agentos/tooling/WindowsSupport.md:9-17 intentionally keeps Agent OS WSL-first while explicitly preserving native-Windows expectations for the Body.
  • learn/agentos/tooling/WindowsSupport.md:28-30 also records the test-unit script's POSIX env-prefix assumption.

The current topology collapses those two support contracts: a Body-only unit test inherits an Agent OS native dependency and cannot start.

The Architectural Reality

  • The unit runner owns two distinct capabilities today:
    1. generic Playwright/Node execution for framework and app unit specs;
    2. an isolated Chroma service required by a bounded subset of Agent OS specs.
  • UNIT_TEST_MODE is already set inside playwright.config.unit.mjs:12; the package-script prefix at package.json:113 is redundant for the canonical config and is non-portable on native Windows shells.
  • Chroma isolation itself remains mandatory for tests that consume it. This ticket changes admission, not the safety contract: Chroma-backed specs must still get a per-run host/port/data-dir and fail closed if that capability cannot start.
  • The owning substrate is the Playwright unit harness under test/playwright/**, with package.json as its human-facing command surface. The AI structure-map sweep found no more appropriate Agent OS service owner; the defect is cross-substrate admission at the test boundary.

The Fix

Make Chroma an explicit/on-demand capability of the unit harness instead of a top-level prerequisite for every spec.

The implementation must preserve the single canonical npm run test-unit -- <spec> entry point. A contributor running a pure Body spec should not need an undocumented flag, a second config, Docker, Python Chroma, or WSL. Agent OS specs that actually require Chroma must continue to acquire the isolated daemon automatically.

Also remove the redundant POSIX UNIT_TEST_MODE=true package-script prefix once the config-owned assignment is proven sufficient on all supported paths.

The exact internal mechanism (fixture/project admission versus another Playwright-native split) is implementation-owned, but it must make the capability boundary explicit and testable rather than parsing filenames ad hoc.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
npm run test-unit -- <spec> package.json:113 + learn/guides/testing/UnitTesting.md One portable command; pure Body specs run without Chroma Fail actionably only when the selected test requires an unavailable capability Unit-testing guide Native Windows x64 Body smoke + existing Linux suite
Unit Chroma lifecycle test/playwright/playwright.config.unit.mjs:14-46 Provision isolated Chroma only for consumers that declare/require it No production reuse; capability startup failure blocks only Chroma-backed tests Config JSDoc/comments Chroma-backed regression spec plus isolation checks
Agent OS Windows boundary learn/agentos/tooling/WindowsSupport.md:9-17 Remains WSL-first; this ticket does not claim native Agent OS support Existing WSL guide Windows support audit Boundary test/documentation review

Acceptance Criteria

  • [L3-deferred — native Windows x64 host needed] On native Windows x64, npm run test-unit -- test/playwright/unit/util/Matrix.spec.mjs reaches and completes the selected pure Body spec without starting or importing the Chroma CLI.
  • The same canonical command remains byte-for-byte valid on Linux/macOS and in CI.
  • A representative Chroma-backed unit slice still provisions a fresh isolated daemon automatically and passes.
  • A falsifier proves a pure Body run does not listen on a Chroma test port and does not create a neo-chroma-unit-test-* data directory.
  • The package script no longer depends on POSIX-only inline environment assignment, with UNIT_TEST_MODE still true inside every unit worker.
  • No test may fall through to production Chroma coordinates or reuse a production collection.
  • The testing guide names the capability boundary: WSL remains the supported path for Agent OS work, while pure Body unit work is native-Windows runnable.

Out of Scope

  • Native Windows support for Memory Core, Knowledge Base, daemons, or the full Agent OS.
  • Fixing or forking upstream Chroma.
  • Adding a broad Windows Agent OS CI matrix.
  • Taking over #15429; Parthiv2005 retains the Matrix-test implementation lane.

Avoided Traps

  • Tell every contributor to use WSL — rejected: that is the documented Agent OS boundary, not the Body unit-test contract.
  • Add a contributor-only no-Chroma command — rejected: two public entry points drift and hide which evidence CI validates.
  • Reuse any running Chroma — rejected: it weakens the existing per-run isolation contract.
  • Select capability by brittle filename substring — rejected: capability ownership must be explicit and mechanically witnessed.

Decision Record impact

none — aligned with the scoped support decision recorded by #10135 and learn/agentos/tooling/WindowsSupport.md; does not amend an ADR.

Related

  • #15429 — community lane where the blocker surfaced
  • #10135 — Agent OS Windows support audit
  • #15221 — prior unit-Chroma lifecycle isolation repair
  • chroma-core/chroma#5188 — upstream Windows x64 CLI defect

Live latest-open sweep: checked latest 20 open issues at 2026-07-19T15:03:47Z; no equivalent found. A2A in-flight claim sweep: checked the latest 30 all-state messages immediately before creation; no overlapping claim found.

Origin Session ID: a0518292-02c3-49ee-af08-adff40bc30b1 Retrieval Hint: "pure Body unit test unconditional Chroma webServer Windows x64 contributor #15429"