Repository navigation
e2e: add multi-node rollout test on EKS - #240
Open
ptalgulk01 wants to merge 1 commit into
Open
ptalgulk01 wants to merge 1 commit into
ptalgulk01 wants to merge 1 commit into
Conversation
Signed-off-by: Prachiti Talgulkar <ptalgulk01@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage so far only exercises a single node per pool, so staged rollout across a fleet has never been tested. TestMidRolloutImageChange does provision two nodes, but it runs on bink only.
TestMultiNodeRollout provisions three nodes in one pool, rolls the pool to a new image, and asserts every node boots the new digest and the pool reports all nodes updated. It declares both bink and eks; the GitHub runners cannot host three nodes, so in practice this provides coverage on EKS.
maxUnavailable is asserted inside the same polling loop that waits for convergence, so the invariant is sampled throughout the rollout. Asserting it once up front would pass trivially, as UpdatingCount is zero before the rollout starts. A violation aborts immediately instead of retrying, which would let a later poll mask it.
A node that stays Degraded past a grace period aborts the wait and reports the daemon's own error. Degraded is cleared on the next successful reconcile, so a retried transient error must not fail the test; without the early abort, a failure such as an unauthorized registry pull surfaced only as an opaque matcher timeout minutes later.
Each node's daemon pod is also checked for Running before waiting for Idle, so a pod stuck in ImagePullBackOff is named directly.