LearnNewsExamplesServices
Frontmatter
id15861
titleRe-land workers:4 in CI with two green full-suite samples
stateClosed
labels
enhancementaitestingperformancebuild
assigneesneo-opus-vega
createdAtJul 24, 2026, 11:30 PM
updatedAtAug 16, 2026, 1:36 AM
githubUrlhttps://github.com/neomjs/neo/issues/15861
authorneo-opus-ada
commentsCount3
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 16, 2026, 1:36 AM

Re-land workers:4 in CI with two green full-suite samples

Closed Backlog/active-chunk-9 enhancementaitestingperformancebuild
neo-opus-ada
neo-opus-ada commented on Jul 24, 2026, 11:30 PM

Context

This is the terminal deliverable of the unit-suite parallelism arc — the flip that actually banks the measured ~2.7× CI wall-clock win. It exists because the re-land currently has no honest home.

It lived as #15790's AC3. Reviewer @neo-kimi-iris flagged the consequence on PR #15846 (Required Action): that PR delivers #15790's AC1 and AC2 but not AC3, so Resolves #15790 would close a ticket whose remaining AC is unshipped future work, stranding the re-land tracker.

She offered two fixes — (a) ResolvesRefs, or (b) move AC3 onto #15847. Both have problems, so this is a third option, filed with the reasoning:

  • (a) is mechanically blocked. agent-pr-body-lint.yml requires Resolves #N on every non-draft agent PR; Refs/Related alone only satisfies it for drafts, and the repo standard is no draft PRs. ResolvesRefs on PR #15846 would trip the body gate.
  • (b) relocates the strand instead of clearing it. #15847's Out of Scope explicitly says "The workers: process.env.CI ? 4 re-land flip + ≥2 green samples — that AC lives on #15790." Moving AC3 there would contradict that line and block PR #15853 (already APPROVED) from closing #15847.

The deeper reason neither works: the re-land is downstream of three enablers (#15789, #15790, #15847), so homing it on any single enabler holds that enabler hostage after its own mechanism work is done. It is its own leaf, and the arc already uses one ticket per deliverable PR (#15783 measurement, #15789 SortZone, #15790 profiling, #15847 genesisProbe). This is the missing leaf, not fragmentation — #15790 AC3 literally specifies "a follow-up probe PR", i.e. a separate PR, which is exactly the 1-PR-per-ticket shape the body lint enforces.

The Problem

#15783 measured a ~2.7× unit-suite wall-clock win at --workers=4. The probe (PR #15784) closed unmerged because parallelism exposed test-isolation defects rather than a bad measurement. Three were found and fixed:

Enabler Defect Status
#15789 SortZone spec leaked a global Neo.applyDeltas override across spec files CLOSED
#15790 Wall-clock profiling budgets bust under CPU contention PR #15846
#15847 genesisProbe child-spawn TOCTOU on ephemeral ports PR #15853 (APPROVED)

With all three landed, the original blocker set is cleared and the flip becomes re-attemptable. What is not yet known is whether three were all of them — parallelism surfaces isolation defects probabilistically, so the re-land is itself the discovery instrument. That is why this ticket's evidence bar is two green samples, not one.

The Architectural Reality

  • test/playwright/playwright.config.unit.mjsworkers: process.env.CI ? 1 : undefined is the line to flip to process.env.CI ? 4 : undefined.
  • The same config carries retries: process.env.CI ? 2 : 0 on the line above. That retry setting is not neutral to this ticket — see AC-2, amended 2026-08-15.
  • The unit-profiling project (from #15790) must keep its dependencies: ['unit'] barrier + workers: 1 serializer, or the budgets bust again the moment the bulk goes wide.
  • unit-brain carries dependencies: ['chroma-setup']; a wide bulk run must not race the Chroma lifecycle.
  • CI single-worker ordering currently masks the whole defect class, so local green is weak evidence — the falsifier only exists in CI at --workers=4.

The Fix

Flip the workers line, run CI, record ≥ 2 zero-retry green full-suite samples at the same head, and record the wall-clock delta against the #15783 baseline.

Acceptance Criteria

  • workers: process.env.CI ? 4 : undefined is live in playwright.config.unit.mjs on dev — delivered in PR #17183's diff; live on dev the moment it merges
  • ≥ 2 green full-suite CI samples at the same head, each with a retry count of ZERO, both linked by run URL. One green run is not evidence against a probabilistic failure mode — and a retried run is not evidence at all. Read the flaky count from the run's own output, never the job's verdict.
  • Measured wall-clock recorded against the #15783 single-worker baseline, stated as observed run times, not a claimed multiplier. A run containing a retry is not a timing sample either — the retry's own wall-clock is inside the total.
  • The unit-profiling barrier + workers: 1 serializer verified still in effect under the wide run (the budgets are the thing parallelism broke first)
  • If a fourth isolation defect surfaces: file it as its own enabler leaf, link it here, and keep the flip off dev until that leaf is closed. A red re-land is a successful probe, not a failed ticket — do not weaken a spec to make the flip stick
  • While any AC-5 leaf is open, the probe PR is not approvable, and the reason is written where the merge gate reads it — its PR body, not an agreement in a thread

Evidence, 2026-08-15 — every AC discharged at head 229754695a (PR #17183)

AC discharged by
1 the config line is in PR #17183's diff; dev carries it on merge
2 two zero-retry samples, same head, no push between: attempt 1 13629 passed · 120 skipped · 0 flaky · attempt 2 identical. The 0 flaky reading carries a positive control — run 31894997372 emits 1 flaky in the same format, so the reporter can see a present case
3 observed, not a multiplier: workers: 1 on dev 14m11s vs workers: 4 6m00s / 6m08s, same 13629 tests, same hour, one variable
4 JSON-report worker assignment from a local wide run: bulk unit on workerIndex {0,1,2,3}, unit-profiling alone on {4}, zero temporal overlap; barrier measured end-to-start (581ms gap), because start-to-start would pass even if the barrier failed
5 two defects surfaced and both are closed — #17186 (PR #17189) and #17192 (PR #17193). Neither spec was weakened; each got its own leaf
6 no AC-5 leaf is open, so the PR is approvable, and its body states that where the merge gate reads it

The two-sample rule is what earned this ticket its outcome, and it is worth recording why. Sample 1 of the very first round was clean; sample 2 at the same head found #17186. The round after that repeated the pattern and found #17192. Both times the job verdict was success and only the retry count separated a clean run from a defect. A one-sample AC would have shipped this flip twice over a live isolation defect.

Amendments

AC-2 and AC-5 amended 2026-08-15 by the ticket author, prompted by @neo-opus-vega's methodology flag before the first sample ran. Original text retained below so nobody re-derives the weaker version.

AC-2 originally read: "≥ 2 green full-suite CI samples at the same head, both linked by run URL — one green run is not evidence against a probabilistic failure mode", with no retry clause.

That version is unfalsifiable for this ticket's own target defect class, and the mechanism is worth stating rather than the conclusion:

  1. Playwright marks a test flaky when it genuinely failed and passed on retry.
  2. Playwright runs the retry in a fresh worker.

Composed: a parallelism-sensitive failure — worker-state pollution or contention timing — is guaranteed to heal on retry, therefore guaranteed to be labelled flaky, therefore guaranteed to be dismissed. The tooling's own recovery mechanism launders precisely the class this flip exists to surface. retries: 2 does not merely weaken AC-2; it makes AC-2 blind to the thing AC-2 is for.

The consequence that changes what you do, not just what you accept: flaky > 0 under workers: 4 is a positive detection, not tolerable noise. It discharges nothing and triggers AC-5. The probe's first sample (run 31892182791) is the anchor — job verdict success, every check green, 13517 passed · 120 skipped · 1 flaky, and a real fourth defect underneath it (#17186).

Retries stay at 2. Requiring a zero-retry sample keeps the probe honest without moving a second variable inside the measurement, so a red remains attributable to parallelism alone.

AC-5 originally read: "revert the flip, file it as its own enabler leaf, and link it here."

The requirement it encodes is that dev stays single-worker while a known defect is open — not the revert command itself. Out of Scope already authorises both paths in one sentence: "fix it in its own leaf, or revert and keep CI honest." The literal revert was written for a flip that had landed; a probe PR that never merged satisfies the same requirement without discarding the instrument, and reverting would re-hide the defect rather than fix it.

AC-6 is new, and it closes a gap the revert used to close for free. "Held open and unmerged" is not equivalent to "reverted": a reverted branch is harmless if merged, whereas a held-open probe PR reports fully green and is one approval from landing. PR #17183 was observed mergeStateStatus: CLEAN with 12/12 checks SUCCESS at 2026-08-15T15:31Z while carrying #17186. The only thing standing between it and dev was that nobody had reviewed it yet. An agreement in a thread is not a merge gate.

Out of Scope

  • Weakening, skipping, or loosening any spec to make the flip pass. If a spec fails under parallelism, the spec or the code has a real isolation defect — fix it in its own leaf, or revert and keep CI honest.
  • Raising workers beyond 4. #15783 measured 4; a different value needs its own measurement.
  • The three landed enablers — #15789, #15790, #15847 own their own mechanisms.
  • Changing retries itself. Whether the suite should retry at all is a real question and a separate one; this ticket only rules that a retried run is not a sample.

Avoided Traps

  • One green sample. The failure mode is non-deterministic; a single green run is exactly the evidence that let PR #15784 look safe before it wasn't. Two samples at the same head is the floor.
  • A green verdict as a green sample. The job says success when a parallelism defect failed once and passed on retry. Read the retry count, not the badge.
  • Local green as proof. CI runs single-worker today, so the defect class is invisible locally — this AC can only be discharged in CI.
  • Treating a red re-land as this ticket failing. The probe working is it surfacing defect #4.
  • Reading "held open, not merged" as equivalent protection to a revert. It is equivalent only until someone approves it.

Related

#15783 (measurement, closed) · #15789 (SortZone, closed) · #15790 (profiling budgets — this ticket takes over its AC3) · #15847 (genesisProbe ports) · #17186 (the fourth isolation defect, surfaced 2026-08-15 — AC-5's leaf) · PR #15846 · PR #15853 · PR #15784 (the closed-unmerged original probe) · PR #17183 (the re-land probe).

Blocked by #17186.

Handoff Retrieval Hints

  • query_raw_memories("unit suite workers 4 re-land two green samples isolation defect probe")
  • query_raw_memories("playwright retry fresh worker flaky launders isolation defect")
  • Prior art: PR #15784 is the closed-unmerged first attempt — read why it closed before re-attempting.

Live latest-open sweep: latest 12 open issues checked at 2026-07-24T21:31Z; no equivalent found (re-land / workers title+body search over 300 open+closed returned only #15783 and #15847). A2A in-flight claim sweep: 12 most recent messages, all read-states — no [lane-claim] on this scope.

tobiu referenced in commit 0c254aa - "fix(test): give the real-tree lint spec headroom over its measured 27s cost (#15878) (#15881) on Jul 25, 2026, 2:41 PM
tobiu unassigned from @neo-opus-ada on Jul 28, 2026, 11:25 AM
tobiu referenced in commit b59631e - "fix(test): the admission pin stops asking the environment to be quiet (#17192) (#17193) on Aug 16, 2026, 12:45 AM
tobiu referenced in commit 66c4ee7 - "test(build): re-land workers:4 in CI as a probe, not a claim (#15861) (#17183) on Aug 16, 2026, 1:36 AM
tobiu closed this issue on Aug 16, 2026, 1:36 AM