LearnNewsExamplesServices
Frontmatter
id12980
titleIntegration CI over-triggers: test.yml scope-classifier runs integration on any non-docs change
stateClosed
labels
enhancementaibuild
assigneesneo-opus-vega
createdAtJun 12, 2026, 11:48 AM
updatedAtJun 12, 2026, 12:08 PM
githubUrlhttps://github.com/neomjs/neo/issues/12980
authorneo-opus-vega
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 12, 2026, 12:08 PM

Integration CI over-triggers: test.yml scope-classifier runs integration on any non-docs change

Closed v13.1.0/archive-v13-1-0-chunk-1 enhancementaibuild
neo-opus-vega
neo-opus-vega commented on Jun 12, 2026, 11:48 AM

Context

Operator-flagged friction (PR #12976): a PR changing ONLY a unit test + its fixture (test/playwright/unit/apps/portal/view/news/discussions/Component.spec.mjs + a .md fixture) triggered the integration CI suite — which makes no sense for a unit-test-only change.

The Problem

.github/workflows/test.yml's scope-classifier (the changes job) computes run_integration as:

const isDocsOnlyPath = file => file.endsWith('.md') || file.startsWith('learn/') || file.startsWith('resources/content/');
const runIntegration = normalizedFiles.some(file => !isDocsOnlyPath(file));

Integration runs for any changed file that isn't docs-only. The default is "run integration UNLESS docs-only" — so every non-docs path-type (a unit-test .mjs, a config, a script) defaults to triggering integration. There is no per-suite path-scoping.

The Fix

Flip run_integration to a whitelist (trigger only on integration-relevant paths):

const runIntegration = normalizedFiles.some(file =>
  file.startsWith('src/') ||                                  // engine the Agent-OS stack runs on
  file.startsWith('ai/') ||                                   // Agent OS — what integration exercises (incl. ai/deploy/docker-compose.test.yml)
  file.startsWith('test/playwright/integration/') ||          // the integration specs + fixtures themselves
  file.startsWith('test/playwright/util/') ||                 // shared test infra used by integration
  file === 'test/playwright/fixtures.mjs' ||
  file === 'test/playwright/setup.mjs' ||
  file === 'test/playwright/playwright.config.integration.mjs' ||
  file === 'package.json' || file === 'package-lock.json' ||  // dep changes can break the integration stack
  file === '.github/workflows/test.yml'                       // CI-logic changes run the full suite to validate
);

The Architectural Reality (why the whitelist must be COMPLETE)

The current blacklist over-triggers — wasteful but safe (never misses). A whitelist's failure mode is the opposite and worse: skip-when-it-was-needed → a real integration regression merges green. So the whitelist must be provably complete. Membership is derived from what the integration suite actually exercises (verified): Agent-OS/cloud integration (KB tenant-isolation, MCP transport, OIDC, heartbeat, backup/restore, multi-tenant ingestion, daemon safety) heavily under ai/, AND it runs the engine — BackupRestoreWipe.integration.spec.mjs imports src/Neo.mjs + src/core/_export.mjs.

Avoided Traps

  • Do NOT use a tight src/+ai/-only whitelist — it would silently SKIP integration on dep bumps (package.json/lock), the integration tests themselves, shared test infra, and CI-logic changes → false-negative CI gaps.
  • ai/deploy/docker-compose.test.yml (the integration Chroma stack) is already under ai/.
  • Preserve the present-but-skipped check pattern (the test matrix job always runs; only inner steps gate) so protected-branch required-checks aren't blocked by a missing check.

Out of Scope (follow-up)

The symmetric rewrite of run_unit into its own per-suite whitelist is the broader root-fix — deferred to bound this change to the operator-reported integration friction. (run_unit over-triggers too, but unit is cheap + over-running is safe.)

Acceptance Criteria

  • run_integration uses the whitelist; a unit-test-only PR (the #12976 shape) does NOT trigger integration.
  • A src/ or ai/ change DOES trigger integration.
  • A dep bump (package.json/lock), an integration-spec change, a shared-infra change, or a test.yml change DOES trigger integration (no false-negative).
  • The integration check remains present-but-skipped (not missing) on skip, preserving protected-branch gating.

Release classification: post-v13 (CI-efficiency; not release-blocking — v13 already shipped).

tobiu closed this issue on Jun 12, 2026, 12:08 PM
tobiu referenced in commit 554d8fb - "fix(ci): scope-classifier triggers integration only on relevant paths (#12980) (#12982) on Jun 12, 2026, 12:08 PM