Context
PR #17183 restored the unit suite's measured workers: 4 speedup while deliberately retaining retries: 2. During that re-land, #17186 and #17192 both surfaced as retry-pass outcomes inside otherwise-successful CI jobs. The temporary two-sample human probe read Playwright's flaky classification; the standing CI contract did not.
Problem
Without a consumer, a test that fails once and passes on retry is reported as flaky but the unit job remains green. Once the two-sample probe ended, the same isolation signal became easy to miss even though Playwright still emitted it.
The detector must remain attributable and resistant to blanket muting: an executed retry-pass is a new defect signal, while a known defect is represented by the existing exact, ticket-owned NEO_TEST_SKIP_CI skip convention.
Prescribed Fix
Use Playwright 1.61.1's existing primitives in test/playwright/playwright.config.unit.mjs:
- Enable
failOnFlakyTests only in CI, so Playwright's own retry-pass classification makes the unit run fail.
- Add Playwright's built-in
github reporter in CI alongside the existing JSON reporter, so the exact flaky test location is annotated while test-results.json remains available.
- Keep
retries: 2 and workers: 4 unchanged.
The existing unit-config contract spec owns the configuration assertions. No workflow parser, log scraper, defect-ledger service, wildcard allowlist, or new ticket is part of this fix.
Acceptance Criteria
Out of Scope
- Changing retry or worker counts.
- Repairing defects the gate finds; each defect remains its own existing-ticket lane.
- Adding a custom parser, workflow step, A2A/defect-ledger transport, or persistent known-flake registry.
Contract Ledger
| Boundary |
Contract |
| Input |
Playwright's per-test retry outcome under CI plus the existing exact NEO_TEST_SKIP_CI skip convention. |
| Output |
Non-zero unit-run status for any executed flaky test, a GitHub annotation naming it, and the existing JSON result artifact. |
| Authority |
Playwright's failOnFlakyTests, github, and json reporter primitives; playwright.config.unit.mjs remains the repository SSOT. |
| Failure states |
Stable pass stays green; retry-pass becomes red/annotated; ticket-owned skip stays skipped; stable failure remains red through the existing path. |
| Invariants |
CI-only gate, retries: 2, workers: 4, no wildcard muting, no second observer pipeline. |
Evidence
L1 mechanism evidence: runs linked from #15861 reported flaky retry-pass outcomes for the real isolation defects #17186 and #17192. Playwright 1.61.1 source/config behavior supplies the native failing and attribution primitives.
Related: #15861 / PR #17183 · #17186 · #17192 · #17180 · #16929
Origin Session ID: 5cd926fa-77e1-4309-8bbf-ca563ab07403
Retrieval hint: query_raw_memories("workers 4 flip retires the flaky detector, green job with a 1 flaky line nobody reads") · original falsification anchor: run 31894997372.
Context
PR #17183 restored the unit suite's measured
workers: 4speedup while deliberately retainingretries: 2. During that re-land, #17186 and #17192 both surfaced as retry-pass outcomes inside otherwise-successful CI jobs. The temporary two-sample human probe read Playwright's flaky classification; the standing CI contract did not.Problem
Without a consumer, a test that fails once and passes on retry is reported as flaky but the unit job remains green. Once the two-sample probe ended, the same isolation signal became easy to miss even though Playwright still emitted it.
The detector must remain attributable and resistant to blanket muting: an executed retry-pass is a new defect signal, while a known defect is represented by the existing exact, ticket-owned
NEO_TEST_SKIP_CIskip convention.Prescribed Fix
Use Playwright 1.61.1's existing primitives in
test/playwright/playwright.config.unit.mjs:failOnFlakyTestsonly in CI, so Playwright's own retry-pass classification makes the unit run fail.githubreporter in CI alongside the existing JSON reporter, so the exact flaky test location is annotated whiletest-results.jsonremains available.retries: 2andworkers: 4unchanged.The existing unit-config contract spec owns the configuration assertions. No workflow parser, log scraper, defect-ledger service, wildcard allowlist, or new ticket is part of this fix.
Acceptance Criteria
failOnFlakyTests; the verdict is driven by Playwright's own flaky outcome, not inferred from the GitHub job verdict.githubreporter names and annotates the exact flaky test; the JSON reporter remains enabled for retained artifacts.skipped; every executed retry-pass is newly flaky and blocks the run without a mutable allowlist.failOnFlakyTestsis absent and red when it is enabled, with direct test attribution.retries: 2and CIworkers: 4remain unchanged.Out of Scope
Contract Ledger
NEO_TEST_SKIP_CIskip convention.failOnFlakyTests,github, andjsonreporter primitives;playwright.config.unit.mjsremains the repository SSOT.retries: 2,workers: 4, no wildcard muting, no second observer pipeline.Evidence
L1 mechanism evidence: runs linked from #15861 reported flaky retry-pass outcomes for the real isolation defects #17186 and #17192. Playwright 1.61.1 source/config behavior supplies the native failing and attribution primitives.
Related: #15861 / PR #17183 · #17186 · #17192 · #17180 · #16929
Origin Session ID: 5cd926fa-77e1-4309-8bbf-ca563ab07403
Retrieval hint:
query_raw_memories("workers 4 flip retires the flaky detector, green job with a 1 flaky line nobody reads")· original falsification anchor: run 31894997372.