Repository navigation
fix(uat): mark RBAC-05 and RBAC-06 flaky in the matrix gate - #97
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #96
Evidence: in the last 17 uat runs, 4 failed the matrix gate with "Expected failures that now PASS" naming exactly RBAC-05 and RBAC-06 (runs 44190279679, 44232635312, 44233018360, 44400202011). The other 13 matched the list (11 failures). Merge_group run 44399609125 and push run 44400202011 on the same commit disagree for the same reason.
Cause: RBAC-05 waits 60s for
.status.endpointUrl, which the operator sets at its next resync (measured 61 to 65s), so it wins that race on some runs. RBAC-06 reuses the pod RBAC-05 creates: with no pod its result is empty ('' does not contain 'not authorized'); with the pod, the operator rejects other-vm and it passes. The two flip together. The suite is upstream's, pinned and unmodified, so there is nothing small to fix in m80's setup.Change:
flaky:lines in uat/expected-failures.txt may pass or fail; the result is printed, not fatal. Every other expected failure is still strict in both directions, and a flaky case that did not run at all still fails the gate. Adds uat/test_assert_matrix.py (5 cases). Other race-group cases (QS-07, NET-01, NET-04, MEM-07) did not flip in these 17 runs and stay strict.Gate: go build, go vet, gofmt, go test -race, python3 -m unittest uat/test_assert_matrix.py.