Skip to content

ci(e2e): run e2e specs with 3 workers on CI - #1486

Merged
gabrielseco merged 3 commits into
ci/e2e-quiet-logsfrom
ci/e2e-playwright-container
Oct 9, 2026
Merged

gabrielseco merged 3 commits into
ci/e2e-quiet-logsfrom
ci/e2e-playwright-container

Conversation

@gabrielseco

@gabrielseco gabrielseco commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

E2E specs run 3 at a time on CI instead of one after another. The e2e job drops from about 2.5 minutes to about 1.5.

Stacked on #1485. Review that first; this PR's net diff is only the worker count.

Why

workers: process.env.CI ? 1 : undefined is the line Playwright's scaffold generates, and it has been there since the original "move to playwright" commit. It doesn't look like a deliberate choice. The setup already supports parallel runs:

  • fullyParallel: true is set.
  • CI runs the browsers against a deployed Vercel preview, so the runner isn't also serving the app.
  • Each spec creates its own employment/contractor with Date.now() names and emails, so specs don't obviously share data.

This PR first tried the official Playwright Docker image as well, to skip the apt install. It didn't help. Across 23 recent runs the apt step took 11–33s (the 2m41s that prompted it was a one-off), and starting the container costs about 28s. On top of that, someone would have to bump the image tag by hand. It's reverted here, so only the worker change remains.

What changed

Toggle details
  • example/playwright.config.ts: CI workers 1 → 3 (ubuntu-latest has 4 vCPUs). Removed the scaffold's "Opt out of parallel tests on CI" comment, which would now be wrong.
  • Local runs are unchanged (undefined = Playwright's default).
  • Branch history: container commit → workers commit (ci(e2e): run e2e specs with 3 workers on CI #1487) → revert of the container commit. Squash merge makes this a single config change.
  • Main risk: sandbox load. 3 specs now seed data against the sandbox at the same time. If seeding gets slow or rate-limited, it would show up as retry # lines in the list output, not as clean failures.
  • No SDK or public API changes.

Screenshots

N/A

Related Resources

Testing

Measured E2E job timings:

Run Job total Tests
#1485, 1 worker 2m35s 1.7m
container + 1 worker 2m34s 1.6m
container + 3 workers 1m36s 40s, no retries
  • 3 workers: all 9 specs pass, no retries or flaky tests
  • Run on this branch (no container): all 9 pass in 39s, no retries
  • Feature flag: N/A

🤖 Generated with Claude Code


Note

Low Risk
Only affects CI test execution parallelism in the example app; no production or SDK behavior changes.

Overview
CI E2E parallelism: On CI, Playwright now runs with 3 workers instead of 1, so test files can run in parallel again rather than being forced serial on the runner.

The old comment about opting out of parallel tests on CI was removed to match the new setting. Local runs are unchanged (workers stays undefined outside CI).

Reviewed by Cursor Bugbot for commit 0949451. Bugbot is set up for automated code reviews on this repo. Configure here.

The install-deps step spent ~2m40s on apt every run because the browser
cache only stores binaries, not OS packages. The mcr.microsoft.com/playwright
image ships both, so the cache, install-deps and install steps go away.
--user 1001 matches the runner's uid, which Firefox (nightly) needs.

The image tag must match @playwright/test in example/package.json; a mismatch
fails fast with Playwright's "Executable doesn't exist" error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Deploy preview for remote-flows ready!

Project:remote-flows
Status: ✅  Deploy successful!
Preview URL:https://remote-flows-aigg7fyt0-remotecom.vercel.app
Latest Commit:0949451

Deployed with vercel-action

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Deploy preview for adp-cost-calculator ready!

Project:adp-cost-calculator
Status: ✅  Deploy successful!
Preview URL:https://adp-cost-calculator-j61alh9zi-remotecom.vercel.app
Latest Commit:0949451

Deployed with vercel-action

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

📊 Coverage Report

⚪ Coverage unchanged

Metric Current Previous Change Status
Lines 89.31% 89.31% 0% ⚪
Statements 88.86% 88.86% 0% ⚪
Functions 88.07% 88.07% 0% ⚪
Branches 80.37% 80.37% 0% ⚪

Detailed Breakdown

Lines Coverage
  • Covered: 5249 / 5877
  • Coverage: 89.31%
  • Change: 0% (0 lines)
Statements Coverage
  • Covered: 5347 / 6017
  • Coverage: 88.86%
  • Change: 0% (0 statements)
Functions Coverage
  • Covered: 1402 / 1592
  • Coverage: 88.07%
  • Change: 0% (0 functions)
Branches Coverage
  • Covered: 3250 / 4044
  • Coverage: 80.37%
  • Change: 0% (0 branches)

✅ Coverage check passed

gabrielseco and others added 2 commits October 8, 2026 19:47
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The container saved nothing: apt install-deps normally takes 11-33s (the
2m41s seen once was an outlier), while initializing the container costs
~28s, and it adds a manually-bumped image tag. Keep only the worker change.

This reverts commit 0db255b.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gabrielseco gabrielseco changed the title ci(e2e): run e2e jobs in the official Playwright container ci(e2e): run e2e specs with 3 workers on CI Oct 8, 2026
@gabrielseco
gabrielseco merged commit 39c93e7 into ci/e2e-quiet-logs Oct 9, 2026
14 checks passed
@gabrielseco
gabrielseco deleted the ci/e2e-playwright-container branch October 9, 2026 10:27
gabrielseco added a commit that referenced this pull request Oct 9, 2026
* ci(e2e): only surface failures in Playwright CI output

Drop DEBUG=pw:api from the PR, main and nightly e2e jobs: it logged every
Playwright action (~4.8k lines per run), including expected waitFor
timeouts that read like failures. On CI, use the dot reporter for a compact
log plus the github reporter so failures show as PR annotations. The html
report is still uploaded, and trace: on-first-retry already records a trace
for every failing test since CI retries twice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(e2e): use the list reporter so each test is named in CI output

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(e2e): run e2e specs with 3 workers on CI (#1486)

* ci(e2e): run e2e jobs in the official Playwright container

The install-deps step spent ~2m40s on apt every run because the browser
cache only stores binaries, not OS packages. The mcr.microsoft.com/playwright
image ships both, so the cache, install-deps and install steps go away.
--user 1001 matches the runner's uid, which Firefox (nightly) needs.

The image tag must match @playwright/test in example/package.json; a mismatch
fails fast with Playwright's "Executable doesn't exist" error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(e2e): run e2e specs with 3 workers on CI (#1487)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* revert: run e2e jobs in the official Playwright container

The container saved nothing: apt install-deps normally takes 11-33s (the
2m41s seen once was an outlier), while initializing the container costs
~28s, and it adds a manually-bumped image tag. Keep only the worker change.

This reverts commit 0db255b.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants