LearnNewsExamplesConnect
Frontmatter
assigneesneo-opus-grace
createdAtNov 5, 2026, 3:23 PM
updatedAtNov 8, 2026, 11:35 PM
githubUrlhttps://github.com/neomjs/neo/issues/20328
authorneo-opus-grace
commentsCount8
parentIssuenull
subIssues
20329 The scope classifier admits e2e tiers for paths they cannot reach
20330 Engine PRs run only the e2e specs their change can reach
20338 Engine PRs start runners only to report a skipped tier
20340 PR Baseline never cancels a run for a superseded head
20344 Review Admission Mergeability spawns a runner per open PR on every dev push
20345 Repo-local lint and check workflows boot up to 13 separate runners per PR
20347 Run units and timezone probes in one prepared CI job
20363 A cancelled dev push leaves its own diff unclassified
20451 The Brain-gated e2e tier retires once no Engine spec needs the bridge
20521 Engine e2e shards split by test count, so one runs past the ceiling
20595 The components tier starts three runners only to report a skip
20597 Guide Lint checks out the Brain to judge an Engine pull request
subIssuesCompleted12
subIssuesTotal12
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtNov 8, 2026, 11:35 PM
milestonev13.4

Engine CI runs each tier only when a change can reach it

Closed v13.4.0/archive-v13-4-0-chunk-3 epicaiperformancebuild
neo-opus-grace
neo-opus-grace commented on Nov 5, 2026, 3:23 PM

Terminal predicate: an Engine PR push holds only the runner slots its diff needs, and no Engine CI job checks out the Brain to build, test or lint against it. One Brain read stays by design: the shared baseline's commit-authorship check reads the team roster module at the Brain's dev (#18757), because the roster decides who the team is and must never come from the pull request. (Scope made precise 2026-11-08, when that read surfaced while retiring the guide lint's Brain checkout under #20597.) Each test tier runs only when the diff can reach it, and the fixed checks run as a handful of jobs, not dozens. Neural Link specs drive the page's own App Worker off-socket (#20370). The bridge and the tool definitions are the Brain's to test, and no CI runs the full Engine-plus-Brain cycle. The dev push runs in full every tier its diff reaches. Jobs and runner-minutes per push both sit measurably below this body's baseline.

Problem scope

The neomjs organization runs on 20 concurrent job slots, and every repository's PRs share them. The operator, 2026-11-05:

  • "otherwise we waste our 20 runners cap and delay team progress waiting on 'on hold' pipelines."
  • "we are limited to 20 runner slots, which makes smarter ci scoping crucial … it is intense with 20+ open PRs."
  • "ORG wide PRs as the limitation."

Engine CI spends that cap as if every change could break everything, and it starts a job for every check.

Baseline. I evaluated the committed classify-test-scope.yml script against the file lists of the 346 PRs merged into dev from 2026-10-13 to 2026-11-05, with the same harness WorkflowScopeClassifier.spec.mjs uses. It admits the full engine e2e tier (4 shards) and the full Brain-gated tier (10 shards) for 279 of them. At the shard times measured on #20321 (engine 3.7 min, Brain 3.9 min), that comes to about 16,200 runner-minutes: 66 % Brain-gated tier, 25 % engine tier, 7 % unit.

The wait. #20321 changed one blog post and its blog.json registration. Each of its ten Brain shards queued 7 to 20 minutes before starting, then ran about 4 minutes. Provisioning is only about 45 seconds of that, so the queue comes from the number of jobs, not from setup.

The job count (2026-11-05, 15:19Z).

  • Open PRs: Engine 15, Brain 2, Institution 1, Skills 2.
  • An Engine head starts 43–78 jobs. A Brain PR starts 34, and a Skills PR 6.
  • Every Engine push also starts about 25 short jobs whatever it changed:
    • the shared PR baseline's ten jobs;
    • three Classify test scope jobs;
    • the lint and helper workflows.
  • In the replay corpus, PRs average 15 test jobs that do work and 2.6 that start a runner only to report a skip.

Every job holds a slot from boot to finish, so short jobs cost slots out of proportion to their seconds.

Three causes:

  1. Misclassification. Paths that no tier can reach still admit tiers: portal content data (6 PRs), buildScripts/ directories outside the e2e launch chain (8), and neo-agent-skills version bumps (12). A change outside the E2E boundary still runs a one-shard Brain smoke (67 PRs), which tests the Brain's dev tip, not the change.
  2. Breadth. A change inside the boundary runs every spec of both tiers. Of the 278 PRs inside it, 97 touch only leaf src/ families (grid 57, dashboard 43, form 13), 19 touch only e2e specs or fixtures, and 29 touch only apps or examples.
  3. Job count. Fixed checks run as separate jobs, and skipped tiers start runners just to say so. The shared baseline does this in every repository.

Why an epic: the three causes need different evidence and different authority:

  • The first is a classifier correction that loses no coverage.
  • The second changes which specs gate a PR before merge. #19655 (D#19599) set that to the full tier for any change inside the boundary, so it needs a dry run and its steward's signal first.
  • The third changes required contexts, and those are the operator's rulesets, in every repository.

Intended solution shape

  • One owner of scope. classify-test-scope.yml keeps deciding every tier. No workflow grows its own paths filter, because a skipped required context strands a PR (neomjs/neo-agent-brain#972).
  • Fail toward running. Every narrowing is a deny-list of paths or packages proven unreachable, so a new directory or dependency still runs every tier.
  • Report, never vanish, never spend a runner on it. GitHub reports a job skipped by if: as Success, and that satisfies a required context. A workflow-level path skip leaves the context pending instead (GitHub's "Using conditions to control job execution"). So a skipped tier skips at job level, and its reason goes in the classifier's summary.
    • Required contexts become one per tier rather than one per shard, so none goes missing.
    • Contexts are the operator's rulesets, so each change ships with its ruleset edit.
  • One job per family of short checks. A family of short checks becomes one job, with one step per check and every failure named, so it holds one slot.
  • The dev push is the safety net. Every tier follows the push's own diff, which for a merge is the merged PR's, and runs its full population whenever that diff reaches it. So a PR-side selection miss (#20330) turns red on the next push, attributed to its merge commit. Until the Brain-gated tier retires (#20370), the classifier also forces it to full on a push. (Amended 2026-11-06 on operator direction: no CI runs the full Engine-plus-Brain cycle. Corrected 2026-11-05 after the #20332 review: an earlier version claimed "every tier in full".)

Out of scope

  • CodeQL and build-all: the dev ruleset requires both, and both are short.
  • The Brain tier's reliability: 48 of 243 completed dev-push runs in the window were red (#17844, #19919).
  • Other repositories' runners, and the PR Baseline queue stall of 2026-11-05.
  • Required-context binding for the lint workflows (#17171).

Avoided traps

  • Workflow-level paths-ignore: strands required contexts.
  • Allow-lists of relevant paths: they go stale silently. The comment on the classifier's E2E predicate records that history.
  • Fewer, longer Brain shards: they save about 45 seconds of provisioning per shard but break the five-minute ceiling set for the e2e tier (#18408).

Decision Record impact: none. The breadth leaf amends #19655's PR-side population, which is a graduated plan, not an ADR. Structure map: N/A. npm run ai:structure-map is Brain-hosted, and neo has no such script. Sweeps at 2026-11-05T14:21Z found no equivalent outcome or prior exclusion:

  • every open epic's terminal predicate (closest: #19655, a coverage outcome);
  • closed epics on "CI runner", "test scope", "pipeline" and "e2e tier";
  • the latest 20 open issues;
  • closed issues on classifier, docs-only and impact boundary, with Out of Scope and Avoided Traps read on #12705, #17853 and #18052;
  • A2A claims since 13:20Z;
  • Memory Core on the problem nouns;
  • my own open assignments.

Origin Session ID: 8545cff9-f4d7-4731-b93e-7dfad1347c8a Retrieval Hint: "Engine CI runner slots classifier e2e Brain tier docs-only blog.json neo-agent-skills baseline 346 PRs"

tobiu referenced in commit dd5fe9e - "docs(roadmap): Engine 13.4 replaces the shipped 13.3 section (#20401) (#20405) on Nov 6, 2026, 10:47 AM