Context
e2e-engine (5/5) on #20514 timed out on all three attempts of run 42318530337: 67 passed (5.2m), then The action 'Run e2e (engine tier)' has timed out after 5 minutes, with zero test failures. The timeouts are correlated, not a flake. The five shards carry nearly the same number of tests and very different amounts of time:
| shard |
tests |
time |
| 1/5 |
68 |
2.9m |
| 2/5 |
70 |
2.4m |
| 3/5 |
69 |
3.5m |
| 4/5 |
66 |
3.7m |
| 5/5 |
67 |
5.2m |
That is 17.7m in total, or 3.5m per shard if balanced. The last full dev run (42320643213) shows the same skew at a smaller scale: 8 shards of 56–67 tests take 2.3–4.0m, 25.0m in total. A grid change whose selection reaches the workstation specs (#20514 changes src/grid/header/plugin/Resizable.mjs) cannot pass the gate, and #20520 is next in line.
The Problem
Two engine-tier rules measure work in units that are not time:
- The split.
test-e2e.yml hands the selection to Playwright's own --shard, which gives every shard the same number of tests. A test costs anything from milliseconds to over a minute, so equal counts are not equal time. On #20514, shard 2/5 ran 4 files (2.3m) and shard 5/5 ran 31 (5.1m), measured from the run's timestamped log.
- The sizing.
tierSelection (from #20330 / PR #20411) sizes a selection's matrix by file count: ceil(8 × selected files / all files). A selection heavier per file than the tier's average gets the full run's files per shard but more time per shard. #20514's selection averages 3.5m per shard against the full run's 3.1m.
The Brain-gated tier had the same defect (#20077) and no longer has it: #20078 balances its shards by measured seconds (brainShard).
The Architectural Reality
buildScripts/util/e2eCiSelection.mjs:731-748: tierSelection measures the Brain tier by seconds and the engine tier by paths => paths.length.
buildScripts/util/e2eCiSelection.mjs:380-405: brainShard assigns files longest-first to the lightest shard, from BRAIN_DURATIONS (e2eBrainDurations.json); a file the table lacks counts as the table's median.
.github/workflows/test-e2e.yml:310-311: npx playwright test … --shard="${SHARD}" ${paths}. The Brain workflow already takes its shard's files from e2eCiSelection.mjs --brain-paths with SHARD set.
- A test's printed duration leaves out its file's hooks and fixtures. #20514 5/5's tests print 4.15m against 5.2m of wall time, and its first test prints 3 ms but ends 15 s into the shard. The time between a job's consecutive test lines in GitHub's timestamped log measures what each file actually cost, and it reproduces every #20514 shard's reported time to within about 0.1m.
The Fix
- Generalize
brainShard into one time-balanced splitter that both tiers call with their own duration table. Do not add a second copy.
- Add
e2eEngineDurations.json: seconds per engine spec file, measured from a full dev run's timestamped log. Add an --engine-durations mode that rebuilds it from gh run view <id> --log output.
tierSelection sizes the engine tier by those seconds.
- Add
--engine-paths, which returns shard SHARD's files of TIER_PATHS. The engine run step passes those files instead of --shard. A dispatch probe still runs the specs it names, as one shard.
Contract Ledger
| Target surface |
Source of authority |
Proposed behavior |
Fallback / edge case |
Docs |
Evidence |
tierSelection engine sizing |
test-e2e.yml header: a shard nearing the ceiling gets a sibling, never a higher ceiling |
shard count from measured seconds, at most FULL_SHARDS.engine |
a file without a measurement counts as the median |
JSDoc |
unit |
e2eCiSelection.mjs --engine-paths |
#20078's --brain-paths |
prints shard SHARD's files of TIER_PATHS, balanced by seconds |
no SHARD: every path |
usage line + JSDoc |
unit |
e2eCiSelection.mjs --engine-durations |
#20078's --brain-durations |
rebuilds the table from a run's timestamped log |
a log without timestamps falls back to the printed durations; skipped tests count nothing |
JSDoc |
unit |
| engine run step |
this ticket |
runs the shard's files, no --shard |
dispatch probe: named specs as given |
workflow comment |
this PR's own e2e-engine run |
Decision Record impact: none (aligned with #20078).
Acceptance Criteria
Out of Scope
- The five-minute ceiling and
FULL_SHARDS.engine (8) stay as they are.
- Refreshing the duration tables automatically in CI.
- The Brain-gated tier's selection, which #20078 already balances.
Avoided Traps
- Another shard. #18895, #19078 and #19843 each added shards. That buys time until the next heavy file lands; the count split stays.
- Sizing by test count. Playwright already balances counts: 66–70 tests per shard on #20514, still 2.4–5.2m.
- Re-running the job. Three attempts on the same head timed out at the same 5.2m.
Related
Parent #20328 · regression from #20330 (PR #20411) · precedent #20077 (PR #20078) · #19843 (PR #19844) · blocks #20514 and possibly #20520 (13.4 gate).
Sweeps
- Live latest-open + closed sweep: latest 20 open (
#20518 down to #19145) plus a closed search for e2e engine shard ceiling (#19843, #20077, #19078, #18895; their fixes read) at 2026-11-07T01:31Z. No equivalent open; no prior exclusion.
- A2A in-flight claim sweep: latest 30 messages, all states, at 01:31Z. No CI or shard claim.
- Memory Core + own-assignment:
query_raw_memories on the shard-ceiling symptom (hits: the #20077 / #20078 trail, Ada's 2026-09-21 shard bump). Own open assignments on this surface: #20328 (parent), #20451 (Brain tier retirement, disjoint).
Origin Session ID: 170dd875-487f-4cd0-9526-9001c436ad39
Retrieval Hint: "engine e2e shard split by measured time e2eCiSelection tierSelection engine-paths durations ceiling 20514"
🖖 Grace (Claude Opus 5.5, Claude Code)
Context
e2e-engine (5/5)on #20514 timed out on all three attempts of run 42318530337:67 passed (5.2m), thenThe action 'Run e2e (engine tier)' has timed out after 5 minutes, with zero test failures. The timeouts are correlated, not a flake. The five shards carry nearly the same number of tests and very different amounts of time:That is 17.7m in total, or 3.5m per shard if balanced. The last full
devrun (42320643213) shows the same skew at a smaller scale: 8 shards of 56–67 tests take 2.3–4.0m, 25.0m in total. A grid change whose selection reaches the workstation specs (#20514 changessrc/grid/header/plugin/Resizable.mjs) cannot pass the gate, and #20520 is next in line.The Problem
Two engine-tier rules measure work in units that are not time:
test-e2e.ymlhands the selection to Playwright's own--shard, which gives every shard the same number of tests. A test costs anything from milliseconds to over a minute, so equal counts are not equal time. On #20514, shard 2/5 ran 4 files (2.3m) and shard 5/5 ran 31 (5.1m), measured from the run's timestamped log.tierSelection(from #20330 / PR #20411) sizes a selection's matrix by file count:ceil(8 × selected files / all files). A selection heavier per file than the tier's average gets the full run's files per shard but more time per shard. #20514's selection averages 3.5m per shard against the full run's 3.1m.The Brain-gated tier had the same defect (#20077) and no longer has it: #20078 balances its shards by measured seconds (
brainShard).The Architectural Reality
buildScripts/util/e2eCiSelection.mjs:731-748:tierSelectionmeasures the Brain tier bysecondsand the engine tier bypaths => paths.length.buildScripts/util/e2eCiSelection.mjs:380-405:brainShardassigns files longest-first to the lightest shard, fromBRAIN_DURATIONS(e2eBrainDurations.json); a file the table lacks counts as the table's median..github/workflows/test-e2e.yml:310-311:npx playwright test … --shard="${SHARD}" ${paths}. The Brain workflow already takes its shard's files frome2eCiSelection.mjs --brain-pathswithSHARDset.The Fix
brainShardinto one time-balanced splitter that both tiers call with their own duration table. Do not add a second copy.e2eEngineDurations.json: seconds per engine spec file, measured from a fulldevrun's timestamped log. Add an--engine-durationsmode that rebuilds it fromgh run view <id> --logoutput.tierSelectionsizes the engine tier by those seconds.--engine-paths, which returns shardSHARD's files ofTIER_PATHS. The engine run step passes those files instead of--shard. A dispatch probe still runs the specs it names, as one shard.Contract Ledger
tierSelectionengine sizingtest-e2e.ymlheader: a shard nearing the ceiling gets a sibling, never a higher ceilingFULL_SHARDS.enginee2eCiSelection.mjs --engine-paths--brain-pathsSHARD's files ofTIER_PATHS, balanced by secondsSHARD: every pathe2eCiSelection.mjs --engine-durations--brain-durations--shardDecision Record impact: none (aligned with #20078).
Acceptance Criteria
e2eEngineDurations.jsonis measured from a fulldevrun, and--engine-durationsrebuilds it from that run's log (unit on a log fixture).devrun's 2.3–4.0m.Out of Scope
FULL_SHARDS.engine(8) stay as they are.Avoided Traps
Related
Parent #20328 · regression from #20330 (PR #20411) · precedent #20077 (PR #20078) · #19843 (PR #19844) · blocks #20514 and possibly #20520 (13.4 gate).
Sweeps
#20518down to#19145) plus a closed search fore2e engine shard ceiling(#19843,#20077,#19078,#18895; their fixes read) at 2026-11-07T01:31Z. No equivalent open; no prior exclusion.query_raw_memorieson the shard-ceiling symptom (hits: the #20077 / #20078 trail, Ada's 2026-09-21 shard bump). Own open assignments on this surface: #20328 (parent), #20451 (Brain tier retirement, disjoint).Origin Session ID: 170dd875-487f-4cd0-9526-9001c436ad39 Retrieval Hint: "engine e2e shard split by measured time e2eCiSelection tierSelection engine-paths durations ceiling 20514"
🖖 Grace (Claude Opus 5.5, Claude Code)