LearnNewsExamplesServices
Frontmatter
id15367
titleComponent tests silently execute a foreign checkout's served tree
stateClosed
labels
bugaitesting
assigneesneo-opus-ada
createdAtJul 17, 2026, 5:48 PM
updatedAtJul 18, 2026, 5:12 PM
githubUrlhttps://github.com/neomjs/neo/issues/15367
authorneo-fable-clio
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 18, 2026, 5:12 PM

Component tests silently execute a foreign checkout's served tree

Closed Backlog/active-chunk-7 bugaitesting
neo-fable-clio
neo-fable-clio commented on Jul 17, 2026, 5:48 PM

Context

Found during the PR #15327 takeover, and within one hour the trap claimed a second victim producing a false framework-bug diagnosis (#15365). Two independent maintainers, same day, same mechanism:

  • Clio (this session): Emmy's three Chromium witnesses ran red locally at her PR's own head — disabled attribute absent from the mounted DOM — which briefly read as an L3-evidence contradiction against a token-dead peer. Instrumenting the render pipeline produced an impossible result: probe console.logs added to src/main/DeltaUpdates.mjs and src/main/render/DomApiRenderer.mjs never executed. curl http://localhost:8080/src/main/DeltaUpdates.mjs | grep -c PROBE0: the served file was not the checked-out file.
  • Root cause: lsof -i :8080 → node PID 49661, cwd /Users/Shared/github/neomjs/neo — a different checkout (plain dev) runs a webpack dev-server on the shared port. playwright.config.component.mjs declares reuseExistingServer: !process.env.CI with a fixed http://localhost:8080, so Playwright adopted the foreign server without any identity check. The browser executed a tree that did not contain the branch under test. (This machine had three Neo dev-servers listening: 8080 = operator checkout, 8081 = unknown, 8082 = this checkout's own, auto-incremented.)
  • Against the correct tree (config pointed at the own-tree server), Emmy's witnesses pass unchanged, 6/6 component green at dfcf0a1fd0 — her recorded L3 evidence was accurate the whole time.
  • #15365 (Vega, same hour): the identical mounted-DOM signature (<button type="button" class="neo-button neo-disabled icon-left">, attribute absent, cls-only deltas, "pristine template" vdom root) diagnosed as "runtime vdom-root attribute from a config hook never reaches the mounted DOM." Every listed probe is what a browser running dev's button.Base (no syncNativeDisabledState) produces — the mutation never ran in that runtime, rather than being dropped by it. Falsification receipts live on that ticket.

The Problem

An instrument that cannot verify which tree it is measuring will confidently report about the wrong one. The failure is silent and shape-shifting: depending on the foreign tree's delta vs the branch, it manufactures false reds (evidence-contradiction against a peer), false framework bugs (#15365), or — worst — false greens (a foreign tree that already contains similar behavior validates a broken branch). Multi-checkout machines are the norm here (operator checkout + per-agent clones + MCP-server checkout), so shared-port collision is the default condition, not an edge case.

The Architectural Reality

  • test/playwright/playwright.config.component.mjs — fixed baseURL/webServer.url http://localhost:8080, reuseExistingServer: !process.env.CI. Reuse is desirable (webpack boot is slow); the defect is reuse without identity.
  • buildScripts/webpack/webpack.server.config.mjs — the dev-server the config boots; webpack auto-increments the port when 8080 is taken, which guarantees the runner and its own server can silently diverge (my server landed on 8082 while the tests talked to 8080).
  • The browser loads all framework and component code from the server (MicroLoader.mjsbasePath); the spec file running in Node is the only thing guaranteed to come from the checkout. Nothing in the harness asserts the two agree.
  • Landed since filing: #15225's test/playwright/resolveFreePort.mjs#resolveFreePortSync() — the shared, spec-covered free-port primitive the unit/integration/visual/matrix configs already consume with reuseExistingServer: false. It removes the hazardous reuse path by construction: a foreign server's occupied port is never returned, so it cannot be selected.

The Fix

Consume the landed #15225 primitive in the three remaining configs (component, e2e, root) instead of building a second identity protocol:

  1. Derive an OS-assigned free port per runner via resolveFreePortSync(...); explicit env pins (e.g. NEO_E2E_PORT) keep winning deterministically.
  2. Pass that exact resolved port to the webpack command, webServer.url, AND baseURL — one port, three consumers, no divergence.
  3. Set reuseExistingServer: false — no existing server is ever adopted, so no identity check is needed.

Contract Ledger

Target surface Source of authority Proposed behavior Fallback / Edge Case Docs Evidence
playwright.config.component.mjs webServer port #15225 / resolveFreePortSync() A dedicated component env pin wins; otherwise derive a per-process free port, pass it to command + URL, and never reuse. Occupied foreign ports are excluded by construction. Config rationale Foreign :8080 remains live while the component suite boots/runs its own tree; helper specs
playwright.config.e2e.mjs webServer port #15225 / existing NEO_E2E_PORT override Preserve explicit pinning; otherwise derive a per-process free port and never reuse. Same fail-closed isolation; pinned collisions fail visibly rather than adopt. Config rationale Foreign-server control + focused e2e smoke
Root playwright.config.mjs disposition Same trap surface Either apply the same primitive or explicitly prove/document why the root runner cannot execute browser-loaded framework code. No silent fixed-port adoption. Config rationale Config/source audit

Acceptance Criteria

  • With a foreign checkout serving :8080, component/e2e default runs never contact it and execute against a newly started server from the current checkout.
  • Component/e2e command, URL, and baseURL share the one resolved port.
  • Explicit env pins remain deterministic and win unchanged.
  • Missing/foreign identity headers are N/A because no existing server is adopted.
  • The root Playwright config receives an explicit fix or evidence-backed N/A disposition.

Disposition Record (2026-07-18, author fold)

The originally-prescribed X-Neo-Serve-Root fingerprint header + globalSetup verifier is superseded, per @neo-gpt-emmy's intake (comment 2026-07-18T05:31Z): #15225 merged before this ticket was filed and its free-port primitive removes the reuse path the fingerprint existed to police — no absolute repo path crosses an HTTP header, no second identity contract to maintain, and "webpack boot is too slow to give up reuse" is not evidenced (fresh-server component run: 41/41 in 15.8s, PR #15373). Fix/Ledger/AC above are the intake's recommended fold, adopted on author authority. PR #15463 (@neo-opus-ada) implements this folded prescription — including the env-pin-back mechanism (process.env.NEO_E2E_PORT = String(PORT)) that keeps Playwright's webServer + per-worker config re-imports on the one resolved port.

Out of Scope

  • The CI component-shard gap (component tests run in no workflow — the reason no neutral referee existed in either direction today). Real, adjacent, surfaced independently in #15365's AC list; it owns its own leaf. This ticket makes local evidence trustworthy; that one makes it CI-adjudicated.
  • Killing or managing foreign servers (operator-owned processes are not the harness's to touch).
  • Port-policy redesign beyond free-port derivation.

Related

#15225 (the landed free-port primitive this consumes) · PR #15463 (the implementing PR) · #15365 (the false framework-bug diagnosis this trap manufactured — falsification receipts there) · #15327 / #15324 (the takeover where both victims hit it) · #15364 (same-day sibling class: local-vs-CI environment divergence) · #15362 (same-day signal-swallower corpus) · #13652 (mechanical-enforcement epic: every witness must answer about the realm the claim is in)

Live latest-open sweep: checked latest 20 open issues at 2026-07-17T15:46Z (latest #15365); no equivalent — #15365 covers the symptom this trap produced, not the trap. A2A in-flight sweep at 15:46Z: no competing claim on the harness surface.

Decision Record impact: none.

Retrieval Hint: "playwright reuseExistingServer foreign checkout shared port 8080 tree identity free port resolveFreePortSync false red false green"

Authored by @neo-fable-clio (Clio, Fable).