Start only the matching e2e suite when its label is added - #837
Start only the matching e2e suite when its label is added#837wallrj-cyberark wants to merge 1 commit into
Conversation
wallrj-cyberark
added
the
kind/cleanup
Verified: the label alone started a runAdding a label to this pull request started a second workflow run, with no push. That is the behaviour that does not happen on
The two runs share the same head SHA, so no commit was pushed between them. Run 2 was created three seconds after I added I used What this run does not proveIt shows the If you would rather see it end to end before merging, say so and I will add |
Adding test-e2e, test-ark or test-ngts to an open pull request currently
does nothing. The workflow uses `on: pull_request: {}`, which takes the
default activity types of opened, synchronize and reopened. There is no
`labeled`, so the label sits on the pull request and no run starts.
Re-running the workflow does not help either, because a re-run replays
the original event payload, which had no labels. The only way through is
to close and reopen the pull request, or push a commit. The failure mode
is silent, so it looks like the job is broken.
Add `labeled` to the activity types, and make each job decide what a
label event means for it:
- verify and test skip, because a label says nothing about the code.
- each e2e job runs only when github.event.label.name, the single
label that was just added, is its own label.
github.event.action is null on push and on workflow_dispatch, so both
guards only ever exclude the label event. Behaviour on push, on
workflow_dispatch and on opened/synchronize/reopened is unchanged, so an
e2e suite whose label is already on the pull request still re-runs on
every new commit.
The trigger itself cannot be filtered by label name, so adding an
unrelated label still creates a workflow run. Every job in it skips, and
a skipped job never claims a runner.
One ordering note: add keep-e2e-cluster before test-e2e, not after.
Adding it afterwards starts nothing, because it matches no job.
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
c83c994 to
cb858f9
Compare
wallrj-cyberark
changed the title
wallrj-cyberark
added
kind/cleanup
Verified on this pull requestRevised after review feedback: an unrelated label used to re-run
Run 1 is the no-regression check: ordinary pushes still behave exactly as before. Run 2 is the fix you asked for. The direct before/after is on the same pull request and the same action — adding Run 3 is the positive half, and the only real proof that I cancelled run 3 about 45 seconds in, once the job selection was visible, to avoid a full 30-minute QA run nobody asked for. It had reached the first seconds of One behaviour change to know aboutAdd Not covered by these runsNo GKE cluster was created at any point, so the |
1 participant
Adding the
test-e2e,test-arkortest-ngtslabel to an open pull request does not start a run. This makes it start one — and only the suite you asked for.Why now?
I hit this on #835. I added the
test-e2elabel, nothing happened, and I had to force-push to fire asynchronizebefore the e2e would start. It looks exactly like a broken job.The cause is that
tests.yamluseson: pull_request: {}, which takes the default activity types.— Events that trigger workflows
labeledis not in that list, so the label sits there and no run starts. Re-running does not help either, because a re-run replays the original event payload, which had no labels.So the label route documented in #833 has always needed a push afterwards. #833 gave us a reliable manual route with
workflow_dispatch, which covers the release case. This fixes the pull request case.What changes
labeledis added to the activity types, and each job then decides what a label event means for it:verifyandtestskip on a label event. A label says nothing about the code, so there is nothing new to verify.github.event.label.name— the single label that was just added — is its own label. Addingtest-ngtsdoes not start the ark or GKE suites.github.event.actionis null onpushand onworkflow_dispatch, so both guards only ever exclude the label event.Nothing else changes. Push,
workflow_dispatchandopened/synchronize/reopenedbehave exactly as they do today, so an e2e suite whose label is already on the pull request still re-runs on every new commit.The one unavoidable wrinkle: GitHub cannot filter the trigger by label name, so adding an unrelated label still creates a workflow run. Every job in it skips, and a skipped job never claims a runner, so it costs nothing but a line in the Actions list.
Full behaviour table — every event this workflow subscribes to
verify,testworkflow_dispatchverify,testtest-e2everify,test,test-e2etest-e2everify,test,test-e2ekind/cleanupaddedtest-e2eaddedtest-e2eonlytest-arkaddedark-test-e2eonlytest-ngtsadded,test-e2ealready onngts-test-e2eonlykeep-e2e-clusteradded,test-e2ealready onThat last row is the one behaviour change worth knowing: add
keep-e2e-clusterbeforetest-e2e, not after. Adding it afterwards starts nothing, because it matches no job. Added first, it is already ingithub.event.pull_request.labelswhen thetest-e2erun starts, so the cleanup step still sees it and keeps the cluster.Verified on this pull request: a matching label starts one job, an unrelated label starts none
For
pull_requestevents GitHub reads the trigger configuration from the merge commit, so this pull request tests its own change. Evidence is in the comments below, since run IDs only exist once the pull request is open.What this does not fix
master, on merge or on a schedule. A greenmastertherefore still does not mean the e2e suites passed.ark-test-e2estill carries itsTEMPORARYcomment about a recurring 400 "conflicting tagging values" error. Whether that flake is still real is unanswered, and it is the thing blocking a nightly run.Both are tracked separately.