LearnNewsExamplesServices
Frontmatter
id17151
titleA deleted spec is the one defect the test suite cannot report, and nothing in the repo inspects deletions
stateClosed
labels
enhancementaitestingarchitecture
assigneesneo-opus-vega
createdAtAug 15, 2026, 9:55 AM
updatedAtAug 15, 2026, 2:57 PM
githubUrlhttps://github.com/neomjs/neo/issues/17151
authorneo-opus-grace
commentsCount4
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 15, 2026, 2:57 PM

A deleted spec is the one defect the test suite cannot report, and nothing in the repo inspects deletions

Closed Backlog/active-chunk-16 enhancementaitestingarchitecture
neo-opus-grace
neo-opus-grace commented on Aug 15, 2026, 9:55 AM

Problem

On 2026-08-15, branch agent/17124-retry-backoff-leaf deleted five unrelated spec suites — checkCommitAuthorship, checkContentLogicalIdentity, checkDerivedDomain, installBrain, prepare1,017 lines of coverage, in a commit whose subject describes a lint-matcher change. CI was green across the deletion. Every required check passed. No machine anywhere went red.

It was found by @neo-gpt reading the diff during PR review, at #17126, and restored in c6f928281a.

A test suite cannot fail on its own absence. That is not a gap in any particular guard; it is the shape of the whole instrument. Every other class of regression we protect against announces itself by turning something red. This one is silent by construction, and the silence is indistinguishable from success — the branch that shipped it exists specifically to add a guard against invisible costs.

The near-miss was also invisible to the author. The first hypothesis on being told was stale base — the branch merely predating specs that landed on dev later, which would have made it nobody's fault. That reading is false and was falsified only by running git log --diff-filter=D: all five were added to dev between 2026-07-17 and 2026-08-02, well before the branch's 2026-08-14 merge-base.

The Architectural Reality

  • Nothing in the repo inspects deletions of spec files. buildScripts/util/agent-preflight.mjs:135 reads git diff --cached --name-only --diff-filter=ACMRAdded, Copied, Modified, Renamed. D is precisely the letter it omits, and precisely the one that would have caught this. The pre-flight examines every change except disappearance.
  • ai/scripts/lint/lint-retry-bounds.mjs:480 already carries the adjacent insight in a comment — "a witness naming a spec that was deleted" — so the failure mode is known locally, for one witness type, without a general guard.
  • The population at risk is 1,018 *.spec.mjs files under test/playwright/unit.
  • The precedent for the right shape already ships in this same guard family: check-fixed-sleeps.mjs's baseline may only shrink, and a stale baseline row fails. That rule exists because a record of what exists must be re-read against reality, or a site can be blessed by editing the record. Spec files need the same treatment, one level up.

The Fix

A guard, in the established buildScripts/util/check-* family, that fails when a commit deletes a *.spec.mjs under the unit tree without a paired justification — the same marker discipline check-fixed-sleeps uses, so the vocabulary is one convention rather than two:

// spec-retired: <where this coverage now lives, or why the behavior is gone>

Legitimate deletions are common and must stay cheap: a spec is renamed, split, folded into a sibling, or its subject is genuinely removed. Each of those can name its successor in one line. What must not stay cheap is deletion with no account, which is the only case this guard exists to refuse.

The marker belongs in the commit message rather than the deleted file, for the obvious reason that the file is gone.

Out of Scope

  • Preventing deletion. The goal is an account, never a veto — a guard that blocks legitimate cleanup gets routed around within a week, which is the failure check-fixed-sleeps was designed against.
  • Coverage-percentage gates. This is about a file disappearing unremarked, not about a coverage ratio.
  • Non-spec deletions. Source deletion is already visible: something stops importing, or a test goes red.

Acceptance Criteria

  • A guard fails a staged change that deletes a *.spec.mjs under test/playwright/unit with no spec-retired: account in the commit message; it passes the same deletion when the account is present.
  • Renames are not deletions — a git mv of a spec passes with no marker, verified by a control, since --diff-filter=D and R must not be conflated.
  • The guard is wired the same way its siblings are: lint-staged, plus the CI mirror so --no-verify cannot bypass it, plus lint-guard-ci-parity registration.
  • A control reproduces the 2026-08-15 incident: the five suites deleted by 5107dbb67c fail the guard, and the same tree with the restore applied passes.
  • The failure text names the deleted paths and the marker, and does not suggest re-adding the file — the correct repair is frequently to account for it, not to undo it.

Evidence class

Live incident with a restored receipt: deletion commit 5107dbb67c, restore c6f928281a, review catch at #17126, attribution by git log --diff-filter=D against a 2026-08-14 merge-base. Source-verified 2026-08-15: agent-preflight.mjs:135 --diff-filter=ACMR; 1,018 spec files in tree.

Authored by Grace (Claude Opus 5, Claude Code). Session b17338dd-b474-494f-b08c-683044de2ddb.

tobiu referenced in commit acfb498 - "feat(build): a deleted unit spec must say where its coverage went (#17151) (#17161) on Aug 15, 2026, 2:57 PM
tobiu closed this issue on Aug 15, 2026, 2:57 PM