Skip to content

Modified supervisor_failure_test for for b/456351949 - #5871

Open
habusa19 wants to merge 9 commits into
openconfig:mainfrom
habusa19:P4RT_supervisor_failure_test
Open

habusa19 wants to merge 9 commits into
openconfig:mainfrom
habusa19:P4RT_supervisor_failure_test

Conversation

@habusa19

@habusa19 habusa19 commented Aug 10, 2026 •

Copy link
Copy Markdown

This test verifies hitless failover and route persistence on dual-supervisor routers by injecting 100 gRIBI routes (50 IPv4 and 50 IPv6), initiating continuous data-plane traffic, and triggering a supervisor switchover via gNOI (SwitchControlProcessor). It ensures that forwarding hardware continues to pass packets with 0% packet loss throughout the outage, confirms that the new primary supervisor accurately updates its OpenConfig redundancy telemetry, and proves the control plane remains healthy by successfully reconnecting and programming 100 additional routes post-switchover without disrupting the existing forwarding table.

@habusa19
habusa19 requested review from a team as code owners August 10, 2026 14:00
@google-cla

google-cla Bot commented Aug 10, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@OpenConfigBot

OpenConfigBot commented Aug 10, 2026 •

Copy link
Copy Markdown

Pull Request Functional Test Report for #5871 / 75bba74

Virtual Devices

Device Test Status Test Documentation Job Raw Log
Arista cEOS status
TE-8.2: success
TE-8.2: Supervisor Failure
dbb55798 Log
Cisco 8000E status
TE-8.2: success
TE-8.2: Supervisor Failure
4412945d Log
Cisco XRd status
TE-8.2: failure
TE-8.2: Supervisor Failure
04a0a938 Log
Juniper ncPTX status
TE-8.2: failure
TE-8.2: Supervisor Failure
dfcbfaf8 Log
Nokia SR Linux status
TE-8.2: failure
TE-8.2: Supervisor Failure
4666621d Log
Openconfig Lemming status
TE-8.2: failure
TE-8.2: Supervisor Failure
2868f152 Log

Hardware Devices

Device Test Status Test Documentation Raw Log
Arista status
TE-8.2: pending authorization
TE-8.2: Supervisor Failure
Cisco status
TE-8.2: pending authorization
TE-8.2: Supervisor Failure
Juniper status
TE-8.2: pending authorization
TE-8.2: Supervisor Failure
Nokia status
TE-8.2: pending authorization
TE-8.2: Supervisor Failure

Help

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the supervisor failure test suite by adding IPv6 support and improving the robustness of FIB validation. It introduces several utility functions to streamline gRIBI operations and telemetry verification, allowing for more granular testing of route persistence during controller switchover events.

Highlights

  • IPv6 Support: Added comprehensive IPv6 support to the supervisor failure test suite, including prefix generation and flow configuration.
  • Helper Function Expansion: Introduced reusable helper functions in internal/gribi and internal/components for bulk gRIBI operations and telemetry validation.
  • Multi-Phase Validation: Updated the test logic to perform multi-phase FIB validation, ensuring routes persist correctly across controller switchovers.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the gRIBI supervisor failure test to support dual-stack (IPv4 and IPv6) validation and adds a post-switchover Phase 2 FIB validation. It refactors prefix installation and verification, moves switchover helpers to a shared package, and adds batch-add and verification methods to the gRIBI client. The review feedback suggests improving test robustness by deferring traffic shutdown, optimizing telemetry verification to run concurrently rather than sequentially, and fixing a potential panic in telemetry validation when a path is not present.

Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
Comment thread internal/components/components.go Outdated
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch 3 times, most recently from 71a7ca3 to e7acb9d Compare August 17, 2026 13:32
@habusa19
habusa19 requested a review from a team as a code owner August 17, 2026 13:32
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch from e7acb9d to 6614912 Compare August 18, 2026 05:10
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch 2 times, most recently from c2edc44 to 07fefb3 Compare August 28, 2026 05:33
@habusa19
habusa19 requested a review from a team as a code owner August 28, 2026 05:33
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch from 07fefb3 to a2b26f6 Compare August 28, 2026 07:42
@habusa19

habusa19 commented Sep 1, 2026

Copy link
Copy Markdown
Author

/Gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the supervisor failure test to support dual-stack (IPv4 and IPv6) traffic, introducing helper functions to generate IP prefixes, appending dual-stack flows, and verifying traffic loss during and after a supervisor switchover. The gRIBI client is also updated to support custom redundancy modes and bulk IP prefix additions. The review feedback highlights critical issues: first, using gnmi.Watch during a supervisor switchover can be unreliable due to gNMI server restarts, so a manual polling loop is recommended; second, the generateIPv4Prefixes and generateIPv6Prefixes helper functions lack nil checks on IP parsing and use unsafe byte incrementing, which can cause runtime panics or incorrect prefix generation.

Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
@ghost

ghost commented Sep 4, 2026

Copy link
Copy Markdown

/fptest virtual

@ghost

ghost commented Sep 4, 2026

Copy link
Copy Markdown

/fptest physical

@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch 4 times, most recently from f5fa1b4 to 0a6d46a Compare September 15, 2026 04:15
Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
@habusa19

Copy link
Copy Markdown
Author

/fptest virtual

@habusa19

Copy link
Copy Markdown
Author

/fptest physical

@yadasandeep-crypto

Copy link
Copy Markdown
Contributor

/fptest juniper-physical

@yadasandeep-crypto

Copy link
Copy Markdown
Contributor

/fptest arista-physical

@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch 2 times, most recently from 15b4c67 to f7605f7 Compare October 8, 2026 18:32
@habusa19

habusa19 commented Oct 8, 2026

Copy link
Copy Markdown
Author

/Gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the supervisor failure test to support dual-stack (IPv4 and IPv6) configurations and validates traffic flow across supervisor switchovers. It also introduces a new deviation to gracefully close the gRIBI client before switchover if required by the platform. Review feedback recommends checking for nil returns from net.ParseIP to avoid runtime panics, adding a sleep delay in the gRIBI reconnection retry loop to prevent a tight loop, and registering clientB cleanup operations using t.Cleanup() after a successful start instead of using defer statements before starting, in alignment with the repository style guide.

Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch 2 times, most recently from 0f154db to ee871eb Compare October 8, 2026 19:12
@habusa19

habusa19 commented Oct 8, 2026

Copy link
Copy Markdown
Author

/Gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the supervisor failure test to support dual-stack (IPv4 and IPv6) traffic and validates both TE-8.2.1 and TE-8.2.2 test cases. It introduces a new deviation gribi_close_client_before_switchover to gracefully close the gRIBI client before a switchover if required by the platform, and adds helper methods to the gRIBI client for batch programming of IPv4 and IPv6 prefixes. The feedback recommends registering a deferred cleanup to stop ATE traffic when starting it, ensuring that traffic does not run indefinitely if the test fails or times out. Comment 1 was removed because it suggests using a manual polling loop with static sleeps, which is a banned anti-pattern under the repository's style guide.

@habusa19

habusa19 commented Oct 8, 2026

Copy link
Copy Markdown
Author

/fptest physical

1 similar comment
@habusa19

habusa19 commented Oct 9, 2026

Copy link
Copy Markdown
Author

/fptest physical

@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch from e178e2a to 5040c15 Compare October 11, 2026 16:29
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch from 5040c15 to aaa1f1b Compare October 11, 2026 16:50
@habusa19

Copy link
Copy Markdown
Author

/Gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the supervisor failure test to support dual-stack (IPv4 and IPv6) traffic flows, introduces the gribi_close_client_before_switchover deviation, and refactors the test logic to improve switchover and telemetry validation. The review feedback recommends avoiding gnmi.Watch with .Await when waiting for a device to boot up after a switchover to prevent connection errors, suggesting a manual polling loop with gnmi.Get instead. Additionally, the reviewer suggests using the modern net/netip package for safer IP address parsing, registering the ATE protocol cleanup immediately after configuration to guarantee execution, and simplifying the client closing logic.

Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
@habusa19

Copy link
Copy Markdown
Author

/fptest physical

@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch from 6ea4180 to 725d241 Compare October 12, 2026 06:49
…into P4RT_supervisor_failure_test

# Conflicts:
#	internal/deviations/deviations.go
#	proto/metadata.proto
#	proto/metadata_go_proto/metadata.pb.go
@habusa19
habusa19 force-pushed the P4RT_supervisor_failure_test branch from a8e420e to af9fb5f Compare October 12, 2026 11:49
@habusa19

Copy link
Copy Markdown
Author

/fptest physical

Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
Comment thread feature/gribi/otg_tests/supervisor_failure_test/supervisor_failure_test.go Outdated
Comment thread proto/metadata.proto Outdated
Comment thread proto/metadata.proto Outdated
@habusa19

Copy link
Copy Markdown
Author

/fptest physical

Comment thread internal/gribi/gribi.go Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants