Skip to content

docs: warn about kernel/driver mismatch on cluster upgrades - #517

Merged
mikemckiernan merged 1 commit into
NVIDIA:mainfrom
nikitabugrovsky:nbugrovs/kernel-upgrade-warning
Oct 6, 2026
Merged

mikemckiernan merged 1 commit into
NVIDIA:mainfrom
nikitabugrovsky:nbugrovs/kernel-upgrade-warning

Conversation

@nikitabugrovsky

@nikitabugrovsky nikitabugrovsky commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Adds an important admonition to openshift/gpu-operator-with-precompiled-drivers.rst warning about GPU driver pod failures caused by kernel/driver version mismatches during OpenShift cluster upgrades.

OpenShift cluster upgrades automatically update node kernel versions. If a cluster upgrades to an RHCOS kernel for which Red Hat has not yet published a matching precompiled driver image in registry.redhat.io/nvidia, GPU driver pods fail to initialize. Recovery is difficult, especially on Secure Boot-enabled nodes or in environments where local driver compilation is disabled. This note gives operators an actionable checklist to avoid the issue before upgrading.

@github-actions

Copy link
Copy Markdown

Documentation preview

https://nvidia.github.io/cloud-native-docs/review/pr-517

@nikitabugrovsky nikitabugrovsky changed the title doc: warn about kernel/driver mismatch on cluster upgrades docs: warn about kernel/driver mismatch on cluster upgrades Sep 30, 2026
@nikitabugrovsky

Copy link
Copy Markdown
Contributor Author

@mikemckiernan could you please have a look at another addition to the precompiled drivers doc?

@mikemckiernan mikemckiernan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usual nits and suggestions. Thanks for updating the docs!

Comment thread openshift/gpu-operator-with-precompiled-drivers.rst Outdated
Comment thread openshift/gpu-operator-with-precompiled-drivers.rst Outdated
Comment thread openshift/gpu-operator-with-precompiled-drivers.rst Outdated
Comment thread openshift/gpu-operator-with-precompiled-drivers.rst Outdated
Add an important admonition to gpu-operator-with-precompiled-drivers.rst
recommending manual upgrade approval, image tag validation, and pinning
driver.version to a fixed tag to avoid GPU driver pods failing after an
OpenShift cluster upgrade to an unsupported RHCOS kernel.

Signed-off-by: Nikita Bugrovsky <nbugrovs@redhat.com>
@nikitabugrovsky
nikitabugrovsky force-pushed the nbugrovs/kernel-upgrade-warning branch from e50b269 to 56f0e1f Compare October 6, 2026 11:28
@nikitabugrovsky

nikitabugrovsky commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author

The usual nits and suggestions. Thanks for updating the docs!

Thank you @mikemckiernan. I went over the file and applied your suggestions. Could you please take a look again?

@mikemckiernan
mikemckiernan merged commit f7cb19d into NVIDIA:main Oct 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants