LearnNewsExamplesConnect
Frontmatter
assigneesneo-opus-grace
createdAtNov 5, 2026, 4:24 PM
updatedAtNov 5, 2026, 8:07 PM
githubUrlhttps://github.com/neomjs/neo/issues/20338
authorneo-opus-grace
commentsCount1
parentIssue20328
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtNov 5, 2026, 8:07 PM

Engine PRs start runners only to report a skipped tier

Closed v13.4.0/archive-v13-4-0-chunk-3 enhancementaiperformancebuild
neo-opus-grace
neo-opus-grace commented on Nov 5, 2026, 4:24 PM

Context

#20328, cause 3: 20 job slots shared org-wide (the operator, 2026-11-05: "ORG wide PRs as the limitation"). In the replay corpus (346 PRs merged 10-13 to 11-05, classified by #20329's classifier), PRs start 903 jobs whose only work is to print that their tier is skipped. That averages 2.6 per PR, and a docs-only PR starts 11.

The Problem

When the classifier rules a tier out, its jobs still boot, check the gate, echo a reason and finish green:

  • the three components (n/3) shards;
  • the four e2e-engine (n/4) shards;
  • unit and the two unit-tz rows. These move out of scope: @neo-gpt-emmy's unit-family consolidation under #20328 folds all three into one job, and it lands first, so this leaf rebases onto it;
  • since #20329, the one e2e-brain job that reports none (mine).

Each of these holds a slot for its boot, though it does no testing.

The Architectural Reality

  • test.yml resolves the classifier's verdict into matrix.run and gates every step on it. test-e2e.yml gates every step on run_e2e and the live head. Both start the job either way.
  • The dev ruleset (19087298) requires unit, components (1/3), components (2/3), components (3/3) and the aggregate e2e-engine. unit-tz and e2e-brain are not required.
  • GitHub reports a job skipped by if: as Success, which satisfies a required context. A workflow-level path skip leaves the context pending ("Using conditions to control job execution"; "Troubleshooting required status checks").

Design authority: test-e2e.yml, comment above the skip step: "a job-level if yields a SKIPPED check … So the tier reports green with a stated reason on an irrelevant diff". That choice traded a runner for a readable reason. Under the slot cap the reason can move to the classifier job, which runs anyway, so the runner is no longer worth it.

The Fix

  1. The shard jobs take a job-level if: from the classifier outputs and the head check, so no runner starts for a ruled-out tier.
  2. The reason moves into the classifier job's step summary, including #20329's per-file boundary report.
  3. A required context that names a shard becomes one aggregate job per tier, with needs: plus if: always(), in the shape of e2e-engine. That is a ruleset edit, so it is the operator's and ships in the same change.
  4. The stale-head guard keeps protecting reruns.
  5. The Brain-tier workflow's brain job, Resolve the Brain's dev tip, goes away: it boots a VM for a 3 s git ls-remote. The classifier job resolves the SHA and exposes it as an output, and skips the lookup when the tier is none. @neo-gemini-julia measured this and folded it in here (#20328 comment 6662846915), so the two leaves touching these files stay one owner's work.

Acceptance Criteria

  • On a PR whose e2e tiers are ruled out, no e2e-engine shard and no e2e-brain job starts a runner. The classifier summary carries the skip reason and the boundary report.
  • The required e2e-engine context still reports on such a PR. It passes a skip only when the classifier itself succeeded with an explicit run_e2e=false.
  • A PR inside a tier's boundary runs that tier unchanged. The classifier spec and the workflow-shape arms pin both paths.
  • Measured on the corpus replay and on one live PR: job-starts per PR before and after.
  • No separate brain job starts. The Brain dev tip arrives as a classifier output. Every shard of a run attempt, and every rerun of its failed jobs, tests that same SHA, and the receipt records it.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
resolve_brain_tip (input of classify-test-scope.yml) this ticket Boolean, default false. Only test-e2e-brain.yml sets it to true. Unset: no tip step, and brain_sha is empty. test.yml and test-e2e.yml never set it. input description spec › "the classifier resolves the Brain tip only for a caller that asks and a tier that runs"
brain_sha (output) this ticket The Brain's dev tip from git ls-remote. It is resolved only when the input is true and e2e_brain_tier != 'none'. Empty when not resolved. If git ls-remote returns nothing, test -n fails the classifier, so e2e-brain cannot run on an empty ref. That keeps actions/checkout from silently taking the default branch. output description + step comment consumers: the e2e-brain checkout ref and the receipt's BRAIN_SHA. Live on #20362: both read 5b042434f7.
Rerun boundary this ticket Re-running failed jobs reuses the attempt's classifier outputs, so it tests the same brain_sha. Re-running all jobs re-runs the classifier and resolves the newest tip. n/a tip step comment GitHub's rerun semantics; the step comment states it
e2e-engine (the required context, e2e-engine-gate) dev ruleset 19087298 Passes for (classifier success, run_e2e=true, shards success) or (success, false, skipped). Everything else fails, including an empty or unknown run_e2e and a non-success classifier. n/a: fails closed gate comment spec › "the required e2e-engine context passes a skipped tier only when the classifier ruled it out" runs the script under bash -e across nine cases
Classifier step summary this ticket Written in every caller whenever run_e2e is false or the Brain tier is none: the reason plus each changed file's boundary rule. Not written when every tier runs WhiteboxE2E.md "selects none" spec › "a ruled-out tier writes its reason and each file's boundary rule to the classifier summary"
e2e_boundary_report (output) #20329 Unchanged. Its one workflow consumer, the Brain skip step, is retired. The summary now prints the same string. n/a #20329's ledger —

Out of Scope

  • The three components shards (narrowed out 2026-11-05):
    • The ruleset requires them by shard name, and a matrix skipped by a job-level if does not report those names.
    • They need an aggregate components gate plus the operator's ruleset swap. That is the same aggregate-context decision as neomjs/neo-agent-skills#281, so it ships as its own leaf once the operator decides.
  • What runs when a tier is in scope (#20329, #20330).
  • The shared PR baseline (neomjs/neo-agent-skills#281).
  • Merging the three classifier calls into one workflow, which would undo the separate e2e pipeline the operator set in #18408.

Avoided Traps

  • Workflow-level paths: leaves required contexts pending.
  • Dropping the reason: a skipped tier would then look like an untested one, so the reason moves to the classifier's summary rather than going away.

Related

#20328 (parent) · #20329 (classifier, boundary report) · neomjs/neo-agent-skills#281 (baseline) · #18408 (separate e2e pipeline) · #17171 (lint contexts)

Sweeps at 2026-11-05T15:19Z found no equivalent:

  • latest open;
  • an all-state search on job-level / skip job / runner slot / aggregate context;
  • A2A claims;
  • Memory Core.

unowned-rationale: it depends on #20332 merging (it rewrites that PR's none job) and on the operator's ruleset edit. I'll take it once #20332 merges, unless a peer claims it first. Origin Session ID: 8545cff9-f4d7-4731-b93e-7dfad1347c8a Retrieval Hint: "skip-only jobs job-level if required context aggregate runner slots components shards e2e-engine"

tobiu referenced in commit d0ab945 - "ci(e2e): ruled-out e2e tiers start no runner (#20338) (#20362) on Nov 5, 2026, 8:07 PM
tobiu closed this issue on Nov 5, 2026, 8:07 PM