Context
The fourth isolation defect, surfaced by the workers: 4 re-land probe on PR #17183 — the enabler leaf #15861's AC-5 asks for.
It is worth stating how it was found, because the finding almost did not survive its own instrument: the CI job reported success. The suite ran green, and only the retry count gave it away.
Running 13638 tests using 4 workers
13517 passed (5.3m) · 120 skipped · 1 flaky ← Retry #1
Under retries: 2 a parallelism-sensitive test that fails once and passes on retry is indistinguishable from a healthy one at the verdict level. #15861's AC-2 asks for "two green full-suite samples", and this sample would have counted.
The Problem
test/playwright/unit/ai/services/memory-core/MailboxService.spec.mjs:1381 — "a marker-cohort change starts a new segment-load generation (#16767)"
for (let turn = 0; turn < 50 && payloadReads < 2; turn++) {
await new Promise(resolve => setImmediate(resolve));
}
expect(payloadReads, 'a marker-only cohort change needs an independent segment result').toBe(2);Error: a marker-only cohort change needs an independent segment result
Expected: 2
Received: 1
Fifty event-loop turns is a guess at how long the second payload read needs. At one worker the guess holds; at four, CPU contention means the second read has not landed when the budget runs out, and the assertion reads a half-finished state as a wrong answer. Nothing about the code under test is wrong — the spec's waiting mechanism is.
This is the same defect class as the fixed-sleep work already in flight (#17124 / #17138): a fixed budget standing in for a condition. A sleep guesses in milliseconds, this guesses in event-loop turns; both bust the moment the machine is busier than the author's was.
@neo-opus-grace named the convergence more precisely, and it changes the fix. A bounded spin is a fixed sleep with its nature hidden, and it fails in the worse direction:
|
when the guess is too short |
how the failure reads |
setTimeout(resolve, 1000) |
wait expires |
like timing — the sleep is two lines up, visible in the diff, announcing itself as a wait |
for (turn = 0; turn < 50 …) |
budget expires, falls through |
like logic — Expected: 2 / Received: 1 with no wait anywhere in view |
The spin converts a timing guess into an assertion failure, so its defect is mis-attributed by construction. The natural diagnosis of this failure is "the second payload read never happens" — a product bug in MailboxService. A sleep is honest about being a wait; a spin launders one into a value comparison. The cost is not the failure, it is the diagnosis the failure invites.
The Architectural Reality
The bounded-spin population is small and enumerable:
| site |
shape |
MailboxService.spec.mjs:1224 |
for (let turn = 0; turn < 3; turn++) |
MailboxService.spec.mjs:1298 |
for (let turn = 0; turn < 3; turn++) |
MailboxService.spec.mjs:1381 |
for (let turn = 0; turn < 50 && payloadReads < 2; turn++) ← the failure |
Five specs use setImmediate spins overall: MailboxService, MemoryService.WriteAhead, RequestContextService, bridgeAutoConnectOrdering, ProviderLaneElectionRunner. Only the three above bound the wait with a turn counter; the rest need reading before any of them is assumed equivalent.
The :1381 site is the least naive of the three — it at least has an exit condition (payloadReads < 2) rather than spinning a flat count. Its defect is that the ceiling is still a guess, and on expiry it falls through to assert on whatever state it happened to reach.
The Fix
A bounded spin must exhaust into an explicit timeout failure that names what it was waiting for. @neo-opus-grace's formulation, and it is strictly smaller than the fix this ticket first proposed — it repairs the mis-attribution even where the spin stays:
timed out waiting for payloadReads >= 2 ← what it should say
Expected: 2 / Received: 1 ← what it says now, and it means something else
Once the two outcomes separate, the second means what it says. Converting the spin into a real condition-wait is the fuller repair and still welcome here, but it is no longer the load-bearing part: the harm is that a timing failure currently arrives dressed as a logic failure.
The load-bearing distinction underneath: the current loop cannot tell "the second read never happened" from "the second read has not happened yet", and it reports both as Received: 1.
The turn < 3 siblings deserve the same reading, but they are not this ticket's failure and must not be swept in without their own evidence — a spin that has never failed may be waiting on something already synchronous.
Acceptance Criteria
Out of Scope
- The
workers: 4 flip itself — #15861 owns it, and this leaf is what its AC-5 prescribes.
- Raising the turn budget. Fifty to five hundred moves the failure to a busier machine; it does not remove a guess.
retries: 2. Whether the suite should retry at all is a separate question, raised on #15861.
- The code under test.
MailboxService's behaviour is not implicated — the spec's waiting mechanism is.
Avoided Traps
- Reading the green verdict as evidence. The job said
success. Only 1 flaky and Retry #1 distinguished this run from a healthy one, and AC-2 as written would have counted it.
- Raising the ceiling. The cheapest repair, and it converts a reproducible finding into a rarer one.
- Sweeping all five spin specs into one fix. Three of the five have never failed; changing them without evidence trades a known-good wait for an unproven one and makes any future regression harder to attribute. @neo-opus-grace measured the sibling case this morning: deleting a sleep took a spec from 6.4s to 12.2s while every assertion stayed green, because injection moved before the next poll watermark. A spin around something synchronous costs nothing and removing it changes nothing; a spin around something asynchronous is load-bearing — and the source cannot tell you which. Only running it can, so an inventory is the correct artifact and a sweep would have been the guess.
- Blaming parallelism. Four workers did not break this spec — they revealed that it was already relying on an idle machine.
- Stretching
check-fixed-sleeps to cover this. @neo-opus-grace's guard keys on a setTimeout with a numeric literal ≥ 1000ms; a spin has no milliseconds at all, so there is no threshold to compare. Matching for loops with an awaiting body would be a different rule — "a bounded wait must fail as a timeout, not as a value" rather than "a fixed second-scale wait must name what it waits for" — and bolting the second onto the first gives one guard two subjects and one confusing failure message. If the rule is worth mechanising it wants its own guard.
Evidence class
L4 — reproduced by CI at --workers=4 on PR #17183, run 31892182791, unit job 95030012579, with the failing assertion, its expected/received pair, and the retry marker in the log.
Related
#15861 (the re-land probe whose AC-5 prescribes this leaf) · PR #17183 · #16767 (the ticket the failing test was written for) · #17124 / #17138 (the fixed-sleep class this shares its shape with) · #15874 (unit-brain order-dependent pollution — related isolation family, different mechanism) · #15789 / #15790 / #15847 (the first three enablers)
Live latest-open sweep at 2026-08-15T15:26Z plus a targeted search for MailboxService / segment-load generation / marker-cohort: no duplicate. Nearest is #15874, a different mechanism in the same family. No competing A2A [lane-claim].
Origin Session ID: 5cd926fa-77e1-4309-8bbf-ca563ab07403
Retrieval Hint: query_raw_memories("MailboxService marker-cohort segment-load turn budget flaky workers:4") · falsification anchor: for (let turn = 0; turn < 50 && payloadReads < 2; turn++) at MailboxService.spec.mjs:1378, asserting toBe(2) on fall-through.
Context
The fourth isolation defect, surfaced by the
workers: 4re-land probe on PR #17183 — the enabler leaf #15861's AC-5 asks for.It is worth stating how it was found, because the finding almost did not survive its own instrument: the CI job reported
success. The suite ran green, and only the retry count gave it away.Under
retries: 2a parallelism-sensitive test that fails once and passes on retry is indistinguishable from a healthy one at the verdict level. #15861's AC-2 asks for "two green full-suite samples", and this sample would have counted.The Problem
test/playwright/unit/ai/services/memory-core/MailboxService.spec.mjs:1381— "a marker-cohort change starts a new segment-load generation (#16767)"for (let turn = 0; turn < 50 && payloadReads < 2; turn++) { await new Promise(resolve => setImmediate(resolve)); } expect(payloadReads, 'a marker-only cohort change needs an independent segment result').toBe(2);Fifty event-loop turns is a guess at how long the second payload read needs. At one worker the guess holds; at four, CPU contention means the second read has not landed when the budget runs out, and the assertion reads a half-finished state as a wrong answer. Nothing about the code under test is wrong — the spec's waiting mechanism is.
This is the same defect class as the fixed-sleep work already in flight (#17124 / #17138): a fixed budget standing in for a condition. A sleep guesses in milliseconds, this guesses in event-loop turns; both bust the moment the machine is busier than the author's was.
@neo-opus-grace named the convergence more precisely, and it changes the fix. A bounded spin is a fixed sleep with its nature hidden, and it fails in the worse direction:
setTimeout(resolve, 1000)for (turn = 0; turn < 50 …)Expected: 2 / Received: 1with no wait anywhere in viewThe spin converts a timing guess into an assertion failure, so its defect is mis-attributed by construction. The natural diagnosis of this failure is "the second payload read never happens" — a product bug in
MailboxService. A sleep is honest about being a wait; a spin launders one into a value comparison. The cost is not the failure, it is the diagnosis the failure invites.The Architectural Reality
The bounded-spin population is small and enumerable:
MailboxService.spec.mjs:1224for (let turn = 0; turn < 3; turn++)MailboxService.spec.mjs:1298for (let turn = 0; turn < 3; turn++)MailboxService.spec.mjs:1381for (let turn = 0; turn < 50 && payloadReads < 2; turn++)← the failureFive specs use
setImmediatespins overall:MailboxService,MemoryService.WriteAhead,RequestContextService,bridgeAutoConnectOrdering,ProviderLaneElectionRunner. Only the three above bound the wait with a turn counter; the rest need reading before any of them is assumed equivalent.The
:1381site is the least naive of the three — it at least has an exit condition (payloadReads < 2) rather than spinning a flat count. Its defect is that the ceiling is still a guess, and on expiry it falls through to assert on whatever state it happened to reach.The Fix
A bounded spin must exhaust into an explicit timeout failure that names what it was waiting for. @neo-opus-grace's formulation, and it is strictly smaller than the fix this ticket first proposed — it repairs the mis-attribution even where the spin stays:
Once the two outcomes separate, the second means what it says. Converting the spin into a real condition-wait is the fuller repair and still welcome here, but it is no longer the load-bearing part: the harm is that a timing failure currently arrives dressed as a logic failure.
The load-bearing distinction underneath: the current loop cannot tell "the second read never happened" from "the second read has not happened yet", and it reports both as
Received: 1.The
turn < 3siblings deserve the same reading, but they are not this ticket's failure and must not be swept in without their own evidence — a spin that has never failed may be waiting on something already synchronous.Acceptance Criteria
MailboxService.spec.mjs:1381exhausts into an explicit timeout failure naming what it awaited (payloadReads >= 2), rather than falling through into a value assertion. This is the load-bearing AC: it separates a timing failure from a logic failure.--workers=4in CI with zero retries, evidenced by the run'sflakycount being 0 rather than by the job's green verdict.turn < 3siblings are read and dispositioned — repaired, or explicitly recorded as waiting on something synchronous. A decision, not a sweep.setImmediate-spin specs are inventoried against the same question, with findings filed separately if any share the shape.Out of Scope
workers: 4flip itself — #15861 owns it, and this leaf is what its AC-5 prescribes.retries: 2. Whether the suite should retry at all is a separate question, raised on #15861.MailboxService's behaviour is not implicated — the spec's waiting mechanism is.Avoided Traps
success. Only1 flakyandRetry #1distinguished this run from a healthy one, and AC-2 as written would have counted it.check-fixed-sleepsto cover this. @neo-opus-grace's guard keys on asetTimeoutwith a numeric literal ≥ 1000ms; a spin has no milliseconds at all, so there is no threshold to compare. Matchingforloops with an awaiting body would be a different rule — "a bounded wait must fail as a timeout, not as a value" rather than "a fixed second-scale wait must name what it waits for" — and bolting the second onto the first gives one guard two subjects and one confusing failure message. If the rule is worth mechanising it wants its own guard.Evidence class
L4 — reproduced by CI at
--workers=4on PR #17183, run 31892182791, unit job 95030012579, with the failing assertion, its expected/received pair, and the retry marker in the log.Related
#15861 (the re-land probe whose AC-5 prescribes this leaf) · PR #17183 · #16767 (the ticket the failing test was written for) · #17124 / #17138 (the fixed-sleep class this shares its shape with) · #15874 (unit-brain order-dependent pollution — related isolation family, different mechanism) · #15789 / #15790 / #15847 (the first three enablers)
Live latest-open sweep at 2026-08-15T15:26Z plus a targeted search for
MailboxService/segment-load generation/marker-cohort: no duplicate. Nearest is #15874, a different mechanism in the same family. No competing A2A[lane-claim].Origin Session ID: 5cd926fa-77e1-4309-8bbf-ca563ab07403
Retrieval Hint:
query_raw_memories("MailboxService marker-cohort segment-load turn budget flaky workers:4")· falsification anchor:for (let turn = 0; turn < 50 && payloadReads < 2; turn++)atMailboxService.spec.mjs:1378, assertingtoBe(2)on fall-through.