Skip to content

Cache the Chromium layer in the visual job and refuse a short matrix - #1205

Open
HugoXK wants to merge 1 commit into
mainfrom
dev3/787-visual-matrix
Open

HugoXK wants to merge 1 commit into
mainfrom
dev3/787-visual-matrix

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Written-by: HugoXK
Author: Fledge Platform Dev-3

Fixes #787

What a user sees differently

Nothing on a console page. The change is inside CI: the visual job stops downloading Chromium on every run, and its job summary now names every measured case with its own result instead of one line for the whole matrix. A reader of a red run sees the page, viewport and scheme that failed; a reader of a green run sees the full table.

What changed

  • .github/workflows/ci.yml: inside the existing visual job (no job added, none moved, the graph is unchanged), a step reads the installed Playwright version from playwright/package.json, and actions/cache (0057852…, v4.3.0) restores ~/.cache/ms-playwright under ${{ runner.os }}-playwright-${{ steps.playwright.outputs.version }}. There is deliberately no restore-keys: a Playwright bump has to miss and reinstall rather than launch an older browser under a newer lockfile. The npx playwright install --with-deps chromium step and npm run visual:matrix stay unconditional, so a hit skips the download only.
  • scripts/visual-matrix.ts: records one case per page/viewport/scheme as it runs (record() takes the failure count around the case), refuses a case whose key was already recorded — two cases share one <width>-<slug>.png, so the second would pass on the first one's picture — and after both locales compares the recorded keys with the promised set (checkCoverage()). A dropped page, width or scheme is a failure with the missing keys named, not a shorter table. The table reaches GITHUB_STEP_SUMMARY in the finally, before the exit code is decided.
  • src/lib/visualMatrix.ts: VISUAL_LOCALES, caseKey, expectedCaseKeys, caseCoverage and summaryMarkdown as pure functions, so the matrix's promise can be tested without a browser.
  • test/: visualJobCeiling.test.ts (new) reads the workflow and the runner; visualMatrix.test.ts pins the promised set and the negatives that are pure functions.
  • .github/agent/action-allowlist.json: adds actions/cache, the action the job now uses.
  • docs/autonomous-dev.md: the visual bullet records the cache, the summary and the coverage refusal.

The three negatives

  1. A broken cell fails ci-ok. The matrix step is unguarded, the runner sets process.exitCode = 1 on any failure, and visual is in the aggregate ci-ok needs: list in ci.yml, which test/ciOk.test.ts already pins against REQUIRED_JOBS.
  2. A stale cache cannot green a failed cell. The cache restores ~/.cache/ms-playwright and nothing else; the install and matrix steps are unconditional; there is no continue-on-error; and a summary row's ok is failures.length === from — the assertion count taken around that case in this process, not a cache result.
  3. A dropped page or a reused baseline cannot pass. checkCoverage() compares recorded keys against expectedCaseKeys(): a missing promised case fails with "promised case(s) never ran", a key recorded twice fails with "case key reused", and caseCoverage() reports missing and duplicated at once, so a duplicate row cannot paper over a dropped page.

Red first

At f1c48ed3, in a clean worktree, with the committed test files byte-identical (sha256 082c9b5548d0bc0c5809af65b62fa7ddcf07edf006bbbfe5d893397d2edb6d4d and cac23dbb69cd099965e53996bb89a730ea9b9d555273a8afa4ab4f82bde16554):

npx tsx --test test/visualJobCeiling.test.ts test/visualMatrix.test.ts
# tests 43, pass 33, fail 10

The ten failures are the new behaviour: no cache step, no summary, no promised set, no coverage check. Four assertions pass on the base by design and are the controls this change must not break: the matrix steps are already unguarded, the runner already sets a red exit code, visual is already in ci-ok's needs, and the base has no restore-keys to remove.

At the tip the same two files are 43/43.

Measurement

Filled from this pull request's own runs (the cold number is the first visual run at this tip, the warm number a re-run of the same job at the same tip):

  • visual cold: pending this run — gating, ≤ 75 s. visual warm: pending this run — gating, ≤ 45 s. Baseline: run 44251917876 at 68 s cold / 55 s.
  • docker on the same runs: pending — recorded, not gating (target ≤ 60 s cold / ≤ 40 s warm).
  • Run ids: pending.

Ran:

  • npm run check at the tip — exit 0, 3277 tests / 3277 pass, build clean
  • npx tsx --test test/visualJobCeiling.test.ts test/visualMatrix.test.ts at the tip — 43/43
  • the same two files at f1c48ed3 — 33/43, the red run above
  • node scripts/risk-gate.mjs --base f1c48ed3 — T3, 7 changed files

Not run:

  • The visual matrix on this host: no Playwright Chromium is installed here; the job's own runs above are the measurement.
  • docker locally: the image build, scan and start run in CI.

Risk and rollback

  • The key is runner.os plus the installed Playwright version. Two lockfile states that resolve to the same Playwright version share one browser build, which is what the key promises; a bump misses and downloads (the cold cap). A miss is a full download because there is no restore-keys.
  • actions/cache is pinned by sha and added to the action allowlist; action-pins and workflow-lint judge it like any other action.
  • The coverage check fails the job when the promised set and the loops drift. That is deliberate: the fix is to update expectedCaseKeys in the same change, not to soften the check.
  • Residual risk: a Chromium build cached under a version that is later yanked upstream would still be reused until Playwright moves; the cache is scoped to one repository and evicted by GitHub on its own schedule.
  • Rollback: revert the commit. The job returns to a full download and the old single-line failure output.

Docs and changelog

docs/autonomous-dev.md updated (the visual bullet).

Changelog-not-needed: this is CI plumbing users cannot see.

Fledge risk tier
base: f1c48ed3513a8180b870fae19fb95d32222da44b
head: HEAD
changed files: 7

T3: .github/agent/action-allowlist.json, .github/workflows/ci.yml, docs/autonomous-dev.md, scripts/visual-matrix.ts
T2: src/lib/visualMatrix.ts
T1: test/visualJobCeiling.test.ts, test/visualMatrix.test.ts
T0: docs/autonomous-dev.md
limits: 40 files / 1200 lines (advisory)

risk-tier: T3; basis: .github/agent/action-allowlist.json

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: keep the console visual matrix fast and deterministic

1 participant