Context
Found while trying to satisfy an acceptance criterion on #17241 — "the workstation's rendered appearance is unchanged, proven against the existing visual baseline, both themes, not asserted." The baseline cannot prove that today, because it is already failing before any change is made.
Live latest-open sweep: checked latest 5 open issues at 2026-08-21T23:2x UTC; gh search issues for visual baseline OR golden OR screenshot regression returned zero. No equivalent exists.
The Problem
8 of 8 visual baselines fail on an unmodified tree. Reproduced on a branch whose only changes are a source comment and a new markdown file — neither can affect rendering.
The cause is not a regression. It is that nothing ever runs this suite:
grep -rln 'visual' .github/workflows/ → no match. No workflow references playwright.config.visual.mjs.- No
package.json script invokes it either — the only visual hit is an unrelated "data-visualization" entry.
It runs solely when a human types npx playwright test --config=test/playwright/playwright.config.visual.mjs.
So the goldens rot by construction. Measured:
|
|
cockpit-default-shell.png cut at |
2026-08-21 12:00 |
| SCSS files changed since |
24 |
| last SCSS commit |
2026-08-21 23:04 |
Four FM styling PRs landed in that window — #17505, #17491, #17464, #17455 — all authored by @neo-fable-clio, all CI-green, because the gate that would have caught a visual delta was not part of CI. This is not her miss; it is unreachable from where she stood.
Why this is worse than an unused test
An unused test is inert. This one is load-bearing in prose: FleetCockpitVisual.spec.mjs describes itself as "the design-gate scope floor", and #17241's AC — written by me — cites the baseline as proof. A gate that is referenced as authority while being unrunnable in practice produces false confidence in exactly the reviews that lean on it.
The harness itself is well built and its globalSetup is the reason this was findable at all:
visual harness: the built theme CSS is OLDER than the newest SCSS source — a baseline over stale artifacts is a poisoned golden.
That guard refuses to run over stale CSS. It is a correct instinct pointed at the artifact — and the same reasoning applies one level up to the goldens, which have no equivalent freshness guard.
The Architectural Reality
test/playwright/playwright.config.visual.mjs — self-contained, has its own webServer, so CI-runnability is not the obstacletest/playwright/visual/globalSetup.mjs:27 — the stale-CSS guardtest/playwright/visual/__screenshots__/ — 8 goldens across 2 specs.github/workflows/ — no reference to any of it
The suite requires node ./buildScripts/build/themes.mjs -f -n -e dev first; the guard enforces that and says so in its message, which is exactly what a CI job would need to do.
The Fix
Order matters, and the second step is not mine to take:
- Put the suite in CI, gated on SCSS/theme changes so it does not run on every PR. Rebuild themes, then run. Without this, any golden refresh rots again within a day.
- Refresh the goldens — @neo-fable-clio's call, not an agent's. Regenerating them accepts four PRs' worth of visual change sight-unseen; only the FM design owner can say which deltas are intended. Doing it as a side effect of a CI ticket would launder that decision.
Sequencing them the other way round just resets the clock.
Acceptance Criteria
Out of Scope
- Refreshing the goldens themselves — a design decision owned by @neo-fable-clio, tracked separately or by her on this ticket.
- Any change to the FM design language or to #17241's promotion.
Avoided Traps
Regenerating the goldens to make the suite green. That is the tempting one-line fix and it silently ratifies whatever the last 24 SCSS files changed. The red is currently the only evidence that anything moved.
Adding CI without a demonstrated failure. A visual job that passes because it compares nothing, or because a threshold is too loose, reproduces the present state with a green badge on top — strictly worse, since the badge suppresses the suspicion.
Related
- #17241 — blocked on this for its visual AC; will verify by computed-style assertion instead, which is the better instrument for a token promotion anyway
- #17505, #17491, #17464, #17455 — the SCSS that landed inside the stale window
- #17499 — same shape one subsystem over: an instrument that cannot discriminate, reported rather than worked around
Retrieval Hint: visual baseline golden stale design gate playwright config.visual not in CI
Origin Session ID: 752da6ac-a6c3-447f-8847-1da4ce49deb8
Decision Record impact: none — CI coverage for an existing suite. Structure-map gate: N/A, no ai/ surface and no new placement.
Context
Found while trying to satisfy an acceptance criterion on #17241 — "the workstation's rendered appearance is unchanged, proven against the existing visual baseline, both themes, not asserted." The baseline cannot prove that today, because it is already failing before any change is made.
Live latest-open sweep: checked latest 5 open issues at 2026-08-21T23:2x UTC;
gh search issuesforvisual baseline OR golden OR screenshot regressionreturned zero. No equivalent exists.The Problem
8 of 8 visual baselines fail on an unmodified tree. Reproduced on a branch whose only changes are a source comment and a new markdown file — neither can affect rendering.
The cause is not a regression. It is that nothing ever runs this suite:
grep -rln 'visual' .github/workflows/→ no match. No workflow referencesplaywright.config.visual.mjs.package.jsonscript invokes it either — the onlyvisualhit is an unrelated"data-visualization"entry.It runs solely when a human types
npx playwright test --config=test/playwright/playwright.config.visual.mjs.So the goldens rot by construction. Measured:
cockpit-default-shell.pngcut atFour FM styling PRs landed in that window — #17505, #17491, #17464, #17455 — all authored by @neo-fable-clio, all CI-green, because the gate that would have caught a visual delta was not part of CI. This is not her miss; it is unreachable from where she stood.
Why this is worse than an unused test
An unused test is inert. This one is load-bearing in prose:
FleetCockpitVisual.spec.mjsdescribes itself as "the design-gate scope floor", and #17241's AC — written by me — cites the baseline as proof. A gate that is referenced as authority while being unrunnable in practice produces false confidence in exactly the reviews that lean on it.The harness itself is well built and its
globalSetupis the reason this was findable at all:That guard refuses to run over stale CSS. It is a correct instinct pointed at the artifact — and the same reasoning applies one level up to the goldens, which have no equivalent freshness guard.
The Architectural Reality
test/playwright/playwright.config.visual.mjs— self-contained, has its ownwebServer, so CI-runnability is not the obstacletest/playwright/visual/globalSetup.mjs:27— the stale-CSS guardtest/playwright/visual/__screenshots__/— 8 goldens across 2 specs.github/workflows/— no reference to any of itThe suite requires
node ./buildScripts/build/themes.mjs -f -n -e devfirst; the guard enforces that and says so in its message, which is exactly what a CI job would need to do.The Fix
Order matters, and the second step is not mine to take:
Sequencing them the other way round just resets the clock.
Acceptance Criteria
resources/scss/**globalSetupguidanceOut of Scope
Avoided Traps
Regenerating the goldens to make the suite green. That is the tempting one-line fix and it silently ratifies whatever the last 24 SCSS files changed. The red is currently the only evidence that anything moved.
Adding CI without a demonstrated failure. A visual job that passes because it compares nothing, or because a threshold is too loose, reproduces the present state with a green badge on top — strictly worse, since the badge suppresses the suspicion.
Related
Retrieval Hint:
visual baseline golden stale design gate playwright config.visual not in CIOrigin Session ID: 752da6ac-a6c3-447f-8847-1da4ce49deb8
Decision Record impact: none — CI coverage for an existing suite. Structure-map gate: N/A, no
ai/surface and no new placement.