Skip to content

fix(cni): Stop a failed ADD from deleting a live pod's veth - #573

Open
privateip wants to merge 2 commits into
mainfrom
fix/issue-544
Open

privateip wants to merge 2 commits into
mainfrom
fix/issue-544

Conversation

@privateip

Copy link
Copy Markdown
Collaborator

Summary

During a rolling restart, a replacement container's ADD takes the veth pair over from its still-terminating predecessor; if that ADD then failed, rollback deleted the pair and left the predecessor's live pod with no interface. A failed ADD now hands a taken-over pair back to its prior owner instead of deleting it, so the live pod's own DEL reclaims it.

Test plan

  • A failed replacement ADD on an existing attachment leaves the predecessor's veth in place, owned by its prior owner
  • Purely created pairs still roll back to deletion
  • Lint, unit, root-gated kernel, and Kind e2e suites pass

Fixes #544

🤖 Generated with Claude Code

During a rolling restart, the replacement container's ADD takes the veth pair over from a still-terminating predecessor under the same deterministic names. If that ADD then failed, rollback deleted the pair: the link now carried the new owner's stamp, so the ownership guard in Delete approved the teardown. A predecessor's pod could still be shown as attached with no interface, and nothing retried, because from CNI's point of view ADD and DEL both succeeded.

The DEL half of this race was already fixed; an ADD still removes whatever holds the attachment. This closes the ADD half by making rollback never delete a pair it did not create. Add now reports whether it created the pair or took one over, and in the takeover case who owned it before, and a failed ADD's rollback hands a taken-over pair back to that owner instead of deleting it, so the live predecessor's own DEL reclaims it as usual.

Key changes:
- Add returns an AddResult carrying Adopted and the prior owner's alias
- RestoreOwner re-stamps that alias on a taken-over pair, a no-op when the veth is already gone
- The resource tracker remembers how the veth step resolved and branches rollback on it: created pairs are deleted, taken-over pairs are handed back

Verification: lint, unit tests, the root-gated kernel tests, and the Kind cluster e2e all pass. Two new root-gated tests reproduce the takeover-then-rollback scenario and pin the restore behavior.

Fixes #544

Co-Authored-By: Claude Code <noreply@anthropic.com>
@privateip
privateip requested a review from a team as a code owner September 16, 2026 22:00
ecv
ecv previously approved these changes Sep 16, 2026
The new root-gated veth tests (TestAddAdoptRollbackDoesNotDeletePriorOwnersVeth,
TestRestoreOwnerNoOpWhenLinkGone) create a real kernel VRF/veth under the
same "abc"/"def" identifiers internal/cni's
TestCmdCheckValidConfigMissingResources asserts are absent. The unittest-root
run executes all root-gated package binaries concurrently against the same
host kernel, so the veth tests' VRF/veth pair could race that CHECK test into
spuriously passing (and a veth Delete could race cnibgp's own "abc"/"def"
device). Give the veth tests distinct identifiers so they no longer create
state that collides with either suite.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restarting a pod on an existing attachment leaves the replacement with no interface

2 participants