Skip to content

test(e2e): behaviour examples in Gherkin run on the PR e2e lane — spike - #6740

Draft
jezekra1 wants to merge 3 commits into
mainfrom
test/bdd-examples-spike
Draft

jezekra1 wants to merge 3 commits into
mainfrom
test/bdd-examples-spike

Conversation

@jezekra1

@jezekra1 jezekra1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The grilling on #6558 decided that Gherkin feature files in the repository are the spec, and that they run on the full install for every pull request.
This spike proves the design on one capability before the backfill and the gates build on it.
It adds an acceptance suite with a feature file, an API and a UI driver, and a standalone first-use journey, and runs it on the e2e lane after the smoke suite.

Reviewer guide

What changes in substance

  • A new package, packages/e2e/acceptance. It depends only on the API contract package and its test tools. check:boundary refuses any other dependency, so a test cannot import server internals. Alternative not taken: tests inside packages/e2e/playwright, which depends on the UI package.
  • Gherkin through playwright-bdd. Feature files generate Playwright tests. Step definitions are one call into a Driver. check:steps fails on a step with no definition.
  • One Driver per surface, one Playwright project per Driver. Every Example runs through the API (examples-api). An Example tagged @ui also runs in the browser (examples-ui). Given steps always use the API.
  • The API driver chats over the ACP relay at /api/agents/<id>/acp, the same path the web UI uses.
  • Accounts per worker slot, one worker. The e2e values provision acceptance-<persona>-<slot> accounts for one slot. (open) This is one owner per worker, not per test: the install cannot create accounts at run time, and every owner gets a VM runner of its own that reserves memory. Two slots left the CI node without memory to boot a machine.
  • Warm mock Agents. A slot boots one mock Agent per persona and keeps it, marked by its description. A Given step renames it and sets its answer as MOCK_DEFAULT_REPLY, then waits until the Agent reports it. The mock starts a new process for each new Session, so a script set before the Session does not reach it. An Example costs seconds, not a machine boot. (open)
  • Run-unique data. Every name and text a feature file writes gets a run suffix, so code that special-cases an Example's literal values fails.
  • Lane wiring. e2e:run runs the acceptance suite after the smoke suite, whatever the smoke suite returned. Either failure fails the run. The job summary times it as its own phase. The report goes into the e2e artifact.
  • The first-use journey replaces the overlapping steps of 03-agent.spec.ts: one-click create, the generated avatar, delete from settings. 03 keeps creating the mock Agent that the chained specs depend on.

Where to look closely

  • The journey's one-click Agent must answer with the mock default reply. 03 never chatted with it. If the Generalist card does not create a mock Agent that answers, the journey step fails in CI.
  • The UI negative check. "Ben does not see Ada's agent" first waits for Ben's own agent to show, so the list has loaded before the absence is checked.
  • Lane time. The issue's budget is 20 minutes. The PR lane took about 14 minutes before this change. The job summary of this PR's run gives the new number.

Ruled out

  • A home-grown steps-as-data runner: it was a framework of our own, and playwright-bdd is a maintained one.
  • Accounts created per test through the Keycloak admin API: it needs the admin password in the test process, and each new account boots an assistant Agent.

Gates you changed

  • 03-agent.spec.ts loses three steps. The journey checks the same behaviour.

Scope

  • packages/e2e/acceptance/: package, feature file, steps, drivers, fixtures, journey, mise tasks (check:boundary, check:steps, run).
  • packages/e2e/playwright/src/tests/smoke/03-agent.spec.ts: three steps removed.
  • helm/values-e2e.yaml: three test accounts.
  • .mise/tasks/e2e/run, .mise/tasks/e2e/timings, .github/workflows/e2e.yml: run, time and upload the suite.
  • docs/architecture/e2e.md: the acceptance suite section.
  • pnpm-workspace.yaml, pnpm-lock.yaml: the new package and playwright-bdd 9.2.1.

Issue

Fixes #6739

Testing

  • mise run check passed on the first commit: all repository checks, the new package's check:tsc, check:lint, check:format, check:boundary and check:steps included.
  • After the second commit, the affected checks passed again: both e2e packages' check:tsc and check:lint, check:steps, and check:comment-types.
  • mise run test passed except one vm-runner test that fails in this sandbox's filesystem; this PR changes no vm-runner file (comment on this PR).
  • Not run locally: the acceptance suite itself. It needs a full k3s install with agent machines. This PR's e2e job is its first run, and its job summary gives the lane time against the 20-minute budget.

Written by this agent — https://dam-dev.apps.dam-cl.fmaas.res.ibm.com/a/agent-e2b379ef193ac667?s=b53079a2-ea7a-4dec-9499-bc4475432465

…on the e2e lane

- packages/e2e/acceptance: feature files run by playwright-bdd through an API and a UI driver, one Playwright project each; @ui examples also run in the browser
- a standalone first-use journey in the browser
- one account set per worker slot, warm mock agents reused across workers, a log bundle per failed test
- e2e:run runs the suite after the smoke suite and times it; the report is uploaded with the e2e artifact

Part of #6739

Signed-off-by: Radek Ježek <43778522+jezekra1@users.noreply.github.com>
…oke spec

- 03-agent no longer creates, checks and deletes a one-click agent; the first-use journey does, and checks its generated avatar
- the chat driver fails as soon as the relay closes the connection
- the e2e page says which runs take the acceptance suite

Part of #6739

Signed-off-by: Radek Ježek <43778522+jezekra1@users.noreply.github.com>
@jezekra1 jezekra1 added the agent/agent-e2b379ef193ac667 Work of the software-developer agent agent-e2b379ef193ac667 label Nov 19, 2026 — with DAM Application

Copy link
Copy Markdown
Contributor Author

Local mise run test on this branch: every suite passed except one test in vm-runner, boot::tests::an_entry_holding_a_locked_file_is_kept_whole.
This pull request changes no file in packages/vm-runner.
The test checks which cache entries a locked file keeps, and it fails in this sandbox's filesystem; CI's build / mise check and the test jobs are the authoritative run.

Written by this agent — https://dam-dev.apps.dam-cl.fmaas.res.ibm.com/a/agent-e2b379ef193ac667?s=b53079a2-ea7a-4dec-9499-bc4475432465

…, on one worker slot

- a Given step sets MOCK_DEFAULT_REPLY and waits for it: the mock starts a new process for a new session, and that process took the default reply over a script set before it
- Playwright traces every context already, so the suite no longer starts its own tracing
- one worker slot: each owner gets a VM runner of its own, and a second slot left the CI node without memory to boot a machine

Part of #6739

Signed-off-by: Radek Ježek <43778522+jezekra1@users.noreply.github.com>

Copy link
Copy Markdown
Contributor Author

The first e2e run of the acceptance suite: the smoke suite passed (37 tests, with the trimmed 03-agent), the acceptance suite passed 2 tests and failed 4.
The lane took 1137 s from job start to the end of the acceptance suite, 274 s of it the acceptance suite, with a 240 s timeout inside.

What failed, and what the push changes:

  • The API example got the mock's default reply, not the scripted one. The mock starts a new process for a new Session, and that process sets its default reply over a script set before it. A Given step now sets the answer as the Agent's MOCK_DEFAULT_REPLY and waits until the Agent reports it.
  • Two UI tests and the journey stopped at tracing.start. Playwright already traces every context a test makes. The suite no longer starts its own tracing.
  • One warm Agent stayed starting for 4 minutes. Its condition said AwaitingAllocation: the node had no memory left for its runner. Every owner gets a VM runner of its own, and two worker slots made six owners beside the smoke suite's Agents. The suite now runs one worker with one slot of three accounts. This is a finding for Epic - Agent-written changes meet the behavior users approved #6558 too: one owner per test, or more parallel workers, costs one runner's memory each.

Written by this agent — https://dam-dev.apps.dam-cl.fmaas.res.ibm.com/a/agent-e2b379ef193ac667?s=b53079a2-ea7a-4dec-9499-bc4475432465

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/agent-e2b379ef193ac667 Work of the software-developer agent agent-e2b379ef193ac667

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Behaviour examples in Gherkin run on the PR e2e lane — spike

1 participant