The unit suite loses one arbitrary test per run and names no cause

Context

Filed on operator escalation while reviewing an unrelated red on PR #17795. The observation is first-hand: I pulled and read each CI job log named below rather than reading a dashboard, because the aggregate view is exactly what makes this pattern invisible.

Between 2026-08-25T17:59Z and 2026-08-26T00:02Z the Tests workflow (unit job) failed seven times across seven different specs, with a green run on either side of the window.

The Problem

Every failing run reports the same tally shape — 1 failed, 1 flaky, ~15,190 passed — and a different casualty each time:

Bracketing green runs: 92942a2ed9 (16:36Z) and ad6765fa8c (23:27Z). Run durations are uniform at ~6–7 minutes across both the failures and the successes, so there is no timeout signature.

Why this is one defect and not seven. The casualties share no subsystem — a drag coordinator in the Engine plane, two wake daemons, a KB maintenance script, a graph synthesizer, an extraction inventory. They span both the [unit] and [unit-brain] projects. Seven unrelated specs each developing an independent intermittent fault inside six hours is a far weaker explanation than one run-level condition selecting an arbitrary victim. Uniform failure shape across unrelated subjects is an infrastructure signature.

The casualty I could read in full is not an assertion failure. agentOsExtractionInventory.spec.mjs:1033 died because a subprocess exited non-zero:

Error: Command failed: git -C /home/runner/work/neo/neo status --porcelain=v1 --untracked-files=all
  at buildInventory (ai/scripts/diagnostics/agentOsExtractionInventory.mjs:1798:48)

That is an environment/process failure wearing a test failure's clothes — and git's stderr was discarded, so the log contains no fatal: line and the actual reason is unrecoverable from the artifact.

Confirmed non-deterministic. Re-running the identical head cf6eabcfc5 with no code change produced 0 failing checks — the agentOsExtractionInventory casualty did not reproduce. The victim is therefore not a property of the spec or the commit, which is what makes quarantining any individual test the wrong remedy.

The cause is unknown and this ticket does not assert one. Two hypotheses were formed and killed during triage, recorded so the next reader does not re-derive them:

• Runner disk exhaustion — falsified. The four ENOSPC: no space left on device lines in the log are injected fixtures from test/playwright/unit/ai/services/github-workflow/PullRequestSyncer.spec.mjs:720 and :801, not a real device condition.
• Introduced by the PR that surfaced it — falsified. unit was success at the immediately preceding head 70c1687e45; the head that failed modifies two files with no git interaction, no new files, and no config; and the casualty spec passes locally at that same head (35/35).

Remaining unexplored candidates, explicitly as hypotheses: resource pressure (file descriptors, memory) selecting a victim; contention between the several diagnostics scripts that shell out to git against the shared working tree while Playwright runs workers in parallel; or worker-level pollution from a spec that monkey-patches global fs handles.

The Architectural Reality

• ai/scripts/diagnostics/agentOsExtractionInventory.mjs:1798 — execFileSync('git', ['-C', projectRoot, 'status', …]) with no stderr capture and no error wrapping. A non-zero exit surfaces as an opaque Command failed: with the diagnostic thrown away.
• At least eight scripts under ai/scripts/ shell out to git against the repository root (lint-skill-manifest, generateRosterOnboarding, revisionConfigDiff, consumerRelevanceCensus, planePlacementCensus, mcpHandlerSignatureCensus, agentOsExtractionInventory, agentOsPlaneBoundaryProof). Playwright executes specs across parallel workers against one working tree.
• The Tests workflow is the merge gate every PR is read against, so this instability taxes every lane in the repository, not one.

The Fix

Diagnosis capability first — the cause cannot be named because the suite currently destroys the evidence:

1. Capture subprocess stderr at the shell-out sites. A non-zero git exit must surface the command, exit code, and stderr. This is the change that converts the next occurrence into evidence instead of another opaque casualty.
2. Determine whether the one-victim-per-run pattern is resource-driven by instrumenting a run (worker count, fd/memory ceiling, git contention) and comparing against the bracketing green runs.
3. Fix the cause once named — deliberately unspecified here rather than guessed.

Item 1 stands on its own merits even if the pattern never recurs.

Acceptance Criteria

• A non-zero exit from any git shell-out in ai/scripts/diagnostics/agentOsExtractionInventory.mjs surfaces the command, exit code, and captured stderr in the thrown error; a fixture forcing a failing git invocation asserts stderr reaches the message.
• The same capture discipline is applied to, or explicitly waived per-site with reason for, the other ai/scripts/ git shell-out sites listed above.
• The seven-run window is characterized with evidence: either the cause is named and cited, or the instrumentation added in AC-2 is shown to capture what a recurrence would need, and this ticket records which.
• The 1 failed / 1 flaky co-occurrence is explained — whether the flaky entry is the same spec on retry or a second victim — since the two together are the pattern's fingerprint.
• No test is quarantined, skipped, or retried away as the remedy; a green achieved by suppressing the victim does not satisfy any AC here.

Out of Scope

• Fixing any individual casualty spec as though it were the defect.
• The e2e plane's separate red — #17596 and #17708 own that; this is unit only.
• Raising retry counts or --retries as a mitigation.
• Any change to the specs' assertions.

Avoided Traps

• Reading the tally instead of the log. 1 failed looks like a small ordinary flake in the aggregate view. The pattern is only visible once you read which test failed across consecutive runs, and it is a different one every time.
• Accepting an injected fixture as a real condition. The ENOSPC lines in these logs are deliberate monkey-patches from a passing syncer spec. Treating them as disk exhaustion produces a confident and completely wrong root cause.
• "Red either way" as a merge licence. A pre-existing red does not license shipping past it; each red still needs a named cause or a dated commit, which is what this ticket exists to produce.
• Quarantining the victim. Because the victim is arbitrary, skipping it moves the failure rather than removing it, and destroys the signal that made the pattern legible.

Decision Record impact

none.

Related

Related: #17596 · #17708 (e2e plane, separate) · PR #17795 (surfaced it) · PR #17794

Freeze note: filed under operator escalation during the D#17782 freeze. This is testing/CI substrate rather than a Brain-plane feature lane; @neo-opus-vega holds the freeze and is notified.

Live latest-open sweep: checked latest 20 open issues at 2026-08-26T00:12Z plus a keyword state=all sweep and an A2A in-flight claim scan over the last 30 messages; no equivalent found.

Origin Session ID: 2aaae0bf-8ed1-4d02-9172-841bca0c2467

Retrieval Hint: query_raw_memories("unit suite one arbitrary casualty per run git status execFileSync stderr discarded") Retrieval Hint: dev heads 60dea8f226..e10713444a, green brackets 92942a2ed9 and ad6765fa8c