LearnNewsExamplesConnect
Frontmatter
assigneesneo-opus-grace
createdAtNov 7, 2026, 2:32 AM
updatedAtNov 7, 2026, 1:18 PM
githubUrlhttps://github.com/neomjs/neo/issues/20521
authorneo-opus-grace
commentsCount0
parentIssue20328
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[ ] 20512 Grid column resize ignores flex columns and leaves GroupRow unprojected during live drag
closedAtNov 7, 2026, 1:18 PM

Engine e2e shards split by test count, so one runs past the ceiling

Closed v13.4.0/archive-v13-4-0-chunk-4 bugaitestingbuild
neo-opus-grace
neo-opus-grace commented on Nov 7, 2026, 2:32 AM

Context

e2e-engine (5/5) on #20514 timed out on all three attempts of run 42318530337: 67 passed (5.2m), then The action 'Run e2e (engine tier)' has timed out after 5 minutes, with zero test failures. The timeouts are correlated, not a flake. The five shards carry nearly the same number of tests and very different amounts of time:

shard tests time
1/5 68 2.9m
2/5 70 2.4m
3/5 69 3.5m
4/5 66 3.7m
5/5 67 5.2m

That is 17.7m in total, or 3.5m per shard if balanced. The last full dev run (42320643213) shows the same skew at a smaller scale: 8 shards of 56–67 tests take 2.3–4.0m, 25.0m in total. A grid change whose selection reaches the workstation specs (#20514 changes src/grid/header/plugin/Resizable.mjs) cannot pass the gate, and #20520 is next in line.

The Problem

Two engine-tier rules measure work in units that are not time:

  1. The split. test-e2e.yml hands the selection to Playwright's own --shard, which gives every shard the same number of tests. A test costs anything from milliseconds to over a minute, so equal counts are not equal time. On #20514, shard 2/5 ran 4 files (2.3m) and shard 5/5 ran 31 (5.1m), measured from the run's timestamped log.
  2. The sizing. tierSelection (from #20330 / PR #20411) sizes a selection's matrix by file count: ceil(8 × selected files / all files). A selection heavier per file than the tier's average gets the full run's files per shard but more time per shard. #20514's selection averages 3.5m per shard against the full run's 3.1m.

The Brain-gated tier had the same defect (#20077) and no longer has it: #20078 balances its shards by measured seconds (brainShard).

The Architectural Reality

  • buildScripts/util/e2eCiSelection.mjs:731-748: tierSelection measures the Brain tier by seconds and the engine tier by paths => paths.length.
  • buildScripts/util/e2eCiSelection.mjs:380-405: brainShard assigns files longest-first to the lightest shard, from BRAIN_DURATIONS (e2eBrainDurations.json); a file the table lacks counts as the table's median.
  • .github/workflows/test-e2e.yml:310-311: npx playwright test … --shard="${SHARD}" ${paths}. The Brain workflow already takes its shard's files from e2eCiSelection.mjs --brain-paths with SHARD set.
  • A test's printed duration leaves out its file's hooks and fixtures. #20514 5/5's tests print 4.15m against 5.2m of wall time, and its first test prints 3 ms but ends 15 s into the shard. The time between a job's consecutive test lines in GitHub's timestamped log measures what each file actually cost, and it reproduces every #20514 shard's reported time to within about 0.1m.

The Fix

  1. Generalize brainShard into one time-balanced splitter that both tiers call with their own duration table. Do not add a second copy.
  2. Add e2eEngineDurations.json: seconds per engine spec file, measured from a full dev run's timestamped log. Add an --engine-durations mode that rebuilds it from gh run view <id> --log output.
  3. tierSelection sizes the engine tier by those seconds.
  4. Add --engine-paths, which returns shard SHARD's files of TIER_PATHS. The engine run step passes those files instead of --shard. A dispatch probe still runs the specs it names, as one shard.

Contract Ledger

Target surface Source of authority Proposed behavior Fallback / edge case Docs Evidence
tierSelection engine sizing test-e2e.yml header: a shard nearing the ceiling gets a sibling, never a higher ceiling shard count from measured seconds, at most FULL_SHARDS.engine a file without a measurement counts as the median JSDoc unit
e2eCiSelection.mjs --engine-paths #20078's --brain-paths prints shard SHARD's files of TIER_PATHS, balanced by seconds no SHARD: every path usage line + JSDoc unit
e2eCiSelection.mjs --engine-durations #20078's --brain-durations rebuilds the table from a run's timestamped log a log without timestamps falls back to the printed durations; skipped tests count nothing JSDoc unit
engine run step this ticket runs the shard's files, no --shard dispatch probe: named specs as given workflow comment this PR's own e2e-engine run

Decision Record impact: none (aligned with #20078).

Acceptance Criteria

  • AC-1: The engine tier's shards are balanced by measured seconds through the same splitter as the Brain tier's. Unit: on #20514's selection, no shard's predicted seconds exceed the mean by more than the longest file.
  • AC-2: A selection's engine matrix is sized by measured seconds. Unit: a selection heavier per file than the tier gets more shards than its file count alone would give (#20514's selection: 6, not 5).
  • AC-3: e2eEngineDurations.json is measured from a full dev run, and --engine-durations rebuilds it from that run's log (unit on a log fixture).
  • AC-4: This PR's own e2e-engine shards all finish under the ceiling, and their spread is narrower than the full dev run's 2.3–4.0m.
  • AC-5 (post-merge): #20514's e2e-engine passes on a head that contains this fix.

Out of Scope

  • The five-minute ceiling and FULL_SHARDS.engine (8) stay as they are.
  • Refreshing the duration tables automatically in CI.
  • The Brain-gated tier's selection, which #20078 already balances.

Avoided Traps

  • Another shard. #18895, #19078 and #19843 each added shards. That buys time until the next heavy file lands; the count split stays.
  • Sizing by test count. Playwright already balances counts: 66–70 tests per shard on #20514, still 2.4–5.2m.
  • Re-running the job. Three attempts on the same head timed out at the same 5.2m.

Related

Parent #20328 · regression from #20330 (PR #20411) · precedent #20077 (PR #20078) · #19843 (PR #19844) · blocks #20514 and possibly #20520 (13.4 gate).

Sweeps

  • Live latest-open + closed sweep: latest 20 open (#20518 down to #19145) plus a closed search for e2e engine shard ceiling (#19843, #20077, #19078, #18895; their fixes read) at 2026-11-07T01:31Z. No equivalent open; no prior exclusion.
  • A2A in-flight claim sweep: latest 30 messages, all states, at 01:31Z. No CI or shard claim.
  • Memory Core + own-assignment: query_raw_memories on the shard-ceiling symptom (hits: the #20077 / #20078 trail, Ada's 2026-09-21 shard bump). Own open assignments on this surface: #20328 (parent), #20451 (Brain tier retirement, disjoint).

Origin Session ID: 170dd875-487f-4cd0-9526-9001c436ad39 Retrieval Hint: "engine e2e shard split by measured time e2eCiSelection tierSelection engine-paths durations ceiling 20514"

🖖 Grace (Claude Opus 5.5, Claude Code)

tobiu referenced in commit 53ae0a3 - "ci(e2e): the engine tier's shards split by measured time, as the Brain tier's do (#20521) (#20522) on Nov 7, 2026, 1:18 PM
tobiu closed this issue on Nov 7, 2026, 1:18 PM