LearnNewsExamplesConnect
Frontmatter
assigneesneo-opus-grace
createdAtNov 5, 2026, 3:24 PM
updatedAtNov 6, 2026, 4:22 PM
githubUrlhttps://github.com/neomjs/neo/issues/20330
authorneo-opus-grace
commentsCount12
parentIssue20328
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtNov 6, 2026, 4:22 PM

Engine PRs run only the e2e specs their change can reach

Closed v13.4.0/archive-v13-4-0-chunk-3 enhancementaiperformancebuild
neo-opus-grace
neo-opus-grace commented on Nov 5, 2026, 3:24 PM

Context

Parent epic #20328 measured the baseline: 279 of the 346 PRs merged into dev from 2026-10-13 to 2026-11-05 ran all 14 e2e shards, about 53 runner-minutes each. Many of them reached far less than that:

  • 97 touched only leaf src/ families;
  • 19 touched only e2e specs or fixtures;
  • 29 touched only apps or examples.

Every one of them still ran every spec. The Brain-gated tier alone is 66 % of PR runner time.

The Problem

Selection inside the E2E boundary is all or nothing. A grid-only change runs the Workstation dock specs, the portal specs and every other spec, on both tiers, before merge. The dev push then runs them all again.

The Architectural Reality

  • buildScripts/util/e2eCiSelection.mjs owns both populations (--paths, --brain-paths). It shards them by measured time and declares BRAIN_SMOKE and the quarantines.
  • A spec reaches source through the app or example it navigates. In test/playwright/e2e, apps/workstation has 98 references, apps/grid-cell-editing 23 and apps/portal 11, and about 20 more apps are referenced.
  • What a dev push runs. The Brain-gated tier always runs in full. The engine tier runs its full population when the push's own diff (before...after) reaches the boundary.
    • The engine tier's dev runs cancel when a newer push lands; the Brain tier's newer push replaces only a queued run.
    • So a merge burst could leave a merged commit's own diff unclassified: comment 6664576222. #20363 closed that before any selection.
  • Reach is per spec. An app-level closure is too coarse: Workstation's reaches 41 % of src/, including grid and dashboard (comment 6661250878). Reach has to be measured per spec.

Design authority: #19655, intended solution (amended 2026-11-05 for #20332): "The full tier runs when the existing E2E impact boundary in classify-test-scope.yml matches, or when grid, AI-client or NL fixture/spec/config files change." This leaf amends that for pull requests only. Its steward is @neo-opus-ada.

The Fix

  1. Map. At classify time, read the committed tree. Each spec maps to what it exercises: the app or example it navigates, narrowed by the classes and views the spec itself names. Each of those maps to its static import closure (import and literal import()). The dry run measures this per-spec map against the app-level fallback.

  2. Select. These run both tiers in full:

    • a changed file under a foundation path: src/core/, src/worker/, src/main/, src/vdom/, src/manager/, src/util/ and top-level src/*.mjs;
    • a changed file the map cannot place;
    • a changed module no static edge names, since a computed import can still load it;
    • a change whose files are all e2e specs, because a selective run can't rule out a cross-spec leak through the shared app heap (receipt v3).

    Otherwise each tier runs the specs whose closure contains a changed file or whose app fetches it, any changed specs, the specs the map cannot place, and BRAIN_SMOKE.

  3. Dry-run before wiring. Replay the 346-PR corpus and report, per PR, the selected spec count and runner-minutes against the full ones. Replay every red Brain or engine run in the window, and report whether the selection would have included its failing spec.

  4. Wire. The selection reaches e2eCiSelection.mjs as a path list, and shards stay balanced by measured time. A dev push gets no per-spec selection: every tier it admits runs its full population.

  5. Own the dev red. A dev red after a selective PR green belongs to the merged PR's author, because their change is the one the selection judged unreachable. They triage first. A chronic flaker on record gets a rerun and a note on its own ticket; anything else gets a fix forward or a revert.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
e2eCiSelection.mjs --select, reading E2E_FILES This ticket's Fix 1–2; selectSpecs, tierSelection, selectionOutput Reads the boundary-admitted files as a JSON array and prints GITHUB_OUTPUT lines. A full selection prints e2e_selection=full and its reasons. A selected one adds run_e2e, e2e_brain_tier (selected or none), e2e_{engine,brain}_{paths,matrix,shards}, skip_reason for a ruled-out tier, and the e2e_selection_report heredoc Invalid JSON throws, so the classify job fails and the gate reds JSDoc of the three functions; WhiteboxE2E.md § Hosted CI E2eCiSelection.spec.mjs
classify-test-scope.yml: input select_e2e; outputs e2e_selection, e2e_{engine,brain}_{paths,matrix,shards}, e2e_selection_report; run_e2e, e2e_brain_tier and skip_reason prefer the select step Fix 2–4 On a pull request with admitted files, a caller that sets select_e2e gets a checkout, Node and --select. Its outputs override the boundary's tier verdicts Without select_e2e, on a push, or with no admitted file, the select step does not run: its outputs are empty, and every admitted tier runs in full The workflow's output descriptions WorkflowScopeClassifier.spec.mjs
test-e2e.yml (engine tier) Fix 4 Takes its matrix from e2e_engine_matrix/_shards and its paths from e2e_engine_paths. Prints the report to the summary and the log Empty outputs mean the full [1..4] matrix and RUN_PATHS. The selection floor runs either way FULL_SHARDS.engine Parity arm "each workflow's full matrix is FULL_SHARDS"
test-e2e-brain.yml (Brain-gated tier) Fix 4 Tier selected. Takes its matrix from e2e_brain_matrix/_shards; BRAIN_SELECTED feeds --brain-paths and the receipt Tiers full/smoke/none behave as before; empty outputs mean the full 10-shard matrix FULL_SHARDS.brain, BRAIN_TIERS.selected The selected-tier and parity arms

Acceptance Criteria

  • A dry-run receipt is posted on this ticket before any workflow change. It shows the runner-minutes saved, and it lists every historical red whose failing spec the selection would have skipped, with the cause of each. No miss may remain unexplained. (Receipts v1 6665204144, v2 6665498902 and v3 6672561395. v3 gives one row per miss, each with an evidenced cause or an explicit unknown plus its change shape.)

  • Four things an app's import closure cannot see must count as foundation, unless the replay shows they are covered: (#20411: FOUNDATION and RUNTIME_LOADING in e2eSpecReach.mjs, pinned by E2eSpecReach.spec.mjs.)

    • main-thread addons, which load by name (mainThreadAddons, Neo.main.addon.*);
    • the other entry points: src/main/**, src/vdom/**, src/worker/**, and the canvas and offscreen workers;
    • styles under resources/scss/**;
    • runtime loading: importPath, loadModule({path}), and lazy module: () => import(...).

    These are @neo-opus-ada's four blind spots.

  • The ticket names who owns a dev red that follows a selective green. By default, that is the merged PR's author. (The Fix, step 5.)

  • @neo-opus-ada (steward of #19655) and one non-Claude family give their signal on the dry run before the workflow change merges. (Ada: 6665817550. Emmy: 6672438194, accepted for v3.)

  • A change under a foundation path, or to an unmapped file, runs both tiers in full. A classifier arm pins each case. (#20411; a module no static edge names runs full too.)

  • A change whose files are all e2e specs runs both tiers in full. A classifier arm pins it. (#20411.)

  • A leaf change runs the mapped specs plus BRAIN_SMOKE. The job summary lists the selected population and the reason for each spec. (The wiring PR proves it at L2. The live summary is post-merge: the first leaf-only pull request after merge shows it, because a workflow-changing PR always selects in full. Residual owner: #20328.)

  • A dev push runs the full population of every tier it admits. (#20411's classifier arm; the merge push 1fe7dcab73 is the first live run.)

  • Each tier's shard matrix is sized to its selection, so the job count falls with it. A fixed matrix keeps every job (Ada, 6665817550). (The wiring PR proves it at L2. The live sized matrix is post-merge: the first leaf-only pull request after merge shows it. Residual owner: #20328.)

  • The replay ships with the wiring PR as a unit arm (Ada, 6665817550). (#20411: 98 runs, 5 recorded misses, each with its cause.)

  • The dry run counts how often a merge burst left a tier unrun for a merged commit (comment 6664576222). The wiring PR either closes that hole or records why the Brain tier's forced full is enough. (Receipt v2, 6665498902: 16 of the 101 cancelled engine-tier pushes whose own diff admitted the tier got no run from the next completed push. #20363 closed the hole for every cancelling workflow (PR #20368, 9b418b4764): a dev push now diffs from its own workflow's last completed run. The Brain-gated tier forces full on every dev push, so its replacement runs always covered a skipped commit.)

  • WhiteboxE2E.md states what a PR runs and what the dev push runs. (#20411, § Hosted CI.)

Out of Scope

  • Misclassified paths: the sibling leaf under #20328.
  • Flake reduction: #17844, #19919.
  • The unit and component tiers.

Avoided Traps

  • Hand-kept tables mapping specs to areas: they rot the way the e2e file list did before the classifier switched to rules.
  • Running the full tier on the dev push only: a red then lands on dev for every agent to trip over. #19655 exists because a regression sat on dev for 14 hours.
  • Selection by coverage instrumentation: it needs instrumented runs to stay current. A static map carries no runtime state.

Related

#20328 (parent) · #19655 and D#19599 · #19657 · #20077 (shards split by measured time) · #20363 (the burst hole) · #20374 / PR #20404 (moves adapter-eligible NL specs into the engine tier; the wiring builds on it)

Sweeps: same as #20328, at 2026-11-05T14:21Z. They found no equivalent ticket and no prior exclusion. Origin Session ID: 8545cff9-f4d7-4731-b93e-7dfad1347c8a Retrieval Hint: "e2e impact selection spec app import closure foundation paths dry run Brain tier PR population dev push full"