Repository navigation
Conversation
…on the e2e lane - packages/e2e/acceptance: feature files run by playwright-bdd through an API and a UI driver, one Playwright project each; @ui examples also run in the browser - a standalone first-use journey in the browser - one account set per worker slot, warm mock agents reused across workers, a log bundle per failed test - e2e:run runs the suite after the smoke suite and times it; the report is uploaded with the e2e artifact Part of #6739 Signed-off-by: Radek Ježek <43778522+jezekra1@users.noreply.github.com>
…oke spec - 03-agent no longer creates, checks and deletes a one-click agent; the first-use journey does, and checks its generated avatar - the chat driver fails as soon as the relay closes the connection - the e2e page says which runs take the acceptance suite Part of #6739 Signed-off-by: Radek Ježek <43778522+jezekra1@users.noreply.github.com>
|
Local Written by this agent — https://dam-dev.apps.dam-cl.fmaas.res.ibm.com/a/agent-e2b379ef193ac667?s=b53079a2-ea7a-4dec-9499-bc4475432465 |
…, on one worker slot - a Given step sets MOCK_DEFAULT_REPLY and waits for it: the mock starts a new process for a new session, and that process took the default reply over a script set before it - Playwright traces every context already, so the suite no longer starts its own tracing - one worker slot: each owner gets a VM runner of its own, and a second slot left the CI node without memory to boot a machine Part of #6739 Signed-off-by: Radek Ježek <43778522+jezekra1@users.noreply.github.com>
|
The first e2e run of the acceptance suite: the smoke suite passed (37 tests, with the trimmed What failed, and what the push changes:
Written by this agent — https://dam-dev.apps.dam-cl.fmaas.res.ibm.com/a/agent-e2b379ef193ac667?s=b53079a2-ea7a-4dec-9499-bc4475432465 |
What and why
The grilling on #6558 decided that Gherkin feature files in the repository are the spec, and that they run on the full install for every pull request.
This spike proves the design on one capability before the backfill and the gates build on it.
It adds an acceptance suite with a feature file, an API and a UI driver, and a standalone first-use journey, and runs it on the e2e lane after the smoke suite.
Reviewer guide
What changes in substance
packages/e2e/acceptance. It depends only on the API contract package and its test tools.check:boundaryrefuses any other dependency, so a test cannot import server internals. Alternative not taken: tests insidepackages/e2e/playwright, which depends on the UI package.playwright-bdd. Feature files generate Playwright tests. Step definitions are one call into a Driver.check:stepsfails on a step with no definition.examples-api). An Example tagged@uialso runs in the browser (examples-ui). Given steps always use the API./api/agents/<id>/acp, the same path the web UI uses.acceptance-<persona>-<slot>accounts for one slot. (open) This is one owner per worker, not per test: the install cannot create accounts at run time, and every owner gets a VM runner of its own that reserves memory. Two slots left the CI node without memory to boot a machine.MOCK_DEFAULT_REPLY, then waits until the Agent reports it. The mock starts a new process for each new Session, so a script set before the Session does not reach it. An Example costs seconds, not a machine boot. (open)e2e:runruns the acceptance suite after the smoke suite, whatever the smoke suite returned. Either failure fails the run. The job summary times it as its own phase. The report goes into the e2e artifact.03-agent.spec.ts: one-click create, the generated avatar, delete from settings. 03 keeps creating the mock Agent that the chained specs depend on.Where to look closely
Ruled out
playwright-bddis a maintained one.Gates you changed
03-agent.spec.tsloses three steps. The journey checks the same behaviour.Scope
packages/e2e/acceptance/: package, feature file, steps, drivers, fixtures, journey, mise tasks (check:boundary,check:steps,run).packages/e2e/playwright/src/tests/smoke/03-agent.spec.ts: three steps removed.helm/values-e2e.yaml: three test accounts..mise/tasks/e2e/run,.mise/tasks/e2e/timings,.github/workflows/e2e.yml: run, time and upload the suite.docs/architecture/e2e.md: the acceptance suite section.pnpm-workspace.yaml,pnpm-lock.yaml: the new package andplaywright-bdd9.2.1.Issue
Fixes #6739
Testing
mise run checkpassed on the first commit: all repository checks, the new package'scheck:tsc,check:lint,check:format,check:boundaryandcheck:stepsincluded.check:tscandcheck:lint,check:steps, andcheck:comment-types.mise run testpassed except onevm-runnertest that fails in this sandbox's filesystem; this PR changes novm-runnerfile (comment on this PR).Written by this agent — https://dam-dev.apps.dam-cl.fmaas.res.ibm.com/a/agent-e2b379ef193ac667?s=b53079a2-ea7a-4dec-9499-bc4475432465