nvmeof: fix CSI node plugin crash on immutable Linux distributions#6165
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adjusts NVMe-oF node plugin initialization to avoid crashing on immutable Linux distributions (e.g., Talos) where NVMe components may be built into the kernel and not available as loadable .ko modules.
Changes:
- Stop explicitly modprobing
nvme_fabrics; only ensurenvme_tcpis loaded. - Validate that the NVMe fabrics framework is operational by checking for
/dev/nvme-fabrics.
gadididi
force-pushed
the
nvmeof/fix_loading_nvme_kmod
branch
from
5879c0f to
818ef22
Compare
nixpanic
approved these changes
Mar 10, 2026
Madhu-1
approved these changes
Mar 10, 2026
Member
|
|
Member
|
@Mergifyio rebase |
On distributions like Talos Linux, NVMe modules are compiled directly into the kernel (CONFIG_NVME_TCP=y, CONFIG_NVME_FABRICS=y) instead of being loadable .ko files. Explicitly calling modprobe on nvme_fabrics fails on these systems since there is no .ko file present, causing the CSI node plugin to crash on startup. Removing nvme_fabrics from the module load list is safe because it is always a dependency of nvme_tcp. On normal distributions modprobe loads it automatically as part of the nvme_tcp dependency chain. On immutable distributions it is already baked into the kernel. We verify the fabrics framework is functional after loading nvme_tcp by checking that /dev/nvme-fabrics exists. This device node is created by the kernel on init regardless of whether nvme_fabrics was loaded as a module or compiled in, making it a reliable indicator that NVMe-oF TCP is ready to use. Signed-off-by: gadi-didi <gadi.didi@ibm.com>
ceph-csi-bot
force-pushed
the
nvmeof/fix_loading_nvme_kmod
branch
from
818ef22 to
94fefd7
Compare
Contributor
✅ Branch has been successfully rebased |
Collaborator
|
/test ci/centos/upgrade-tests-cephfs |
Collaborator
|
/test ci/centos/k8s-e2e-external-storage/1.35 |
Collaborator
|
/test ci/centos/k8s-e2e-external-storage/1.34 |
Collaborator
|
/test ci/centos/upgrade-tests-rbd |
Collaborator
|
/test ci/centos/mini-e2e-helm/k8s-1.35 |
Collaborator
|
/test ci/centos/mini-e2e-helm/k8s-1.34 |
Collaborator
|
/test ci/centos/k8s-e2e-external-storage/1.33 |
Collaborator
|
/test ci/centos/mini-e2e/k8s-1.35 |
Collaborator
|
/test ci/centos/mini-e2e/k8s-1.34 |
Collaborator
|
/test ci/centos/mini-e2e-helm/k8s-1.33 |
Collaborator
|
/test ci/centos/mini-e2e/k8s-1.33 |
6 tasks
nixpanic
added
the
ci/skip/multi-arch-build
Contributor
Merge Queue Status
This pull request spent 10 seconds in the queue, with no time running CI. Required conditions to merge
|
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On distributions like Talos Linux, NVMe modules are compiled directly into the kernel (CONFIG_NVME_TCP=y, CONFIG_NVME_FABRICS=y) instead of being loadable .ko files.
Explicitly calling modprobe on nvme_fabrics fails on these systems since there is no .ko file present,
causing the CSI node plugin to crash on startup.
Removing nvme_fabrics from the module load list is safe because it is always a dependency of nvme_tcp.
On normal distributions modprobe loads it automatically as part of the nvme_tcp dependency chain. On immutable distributions it is already baked into the kernel.
We verify the fabrics framework is functional after loading nvme_tcp by checking that
/dev/nvme-fabrics exists.
This device node is created by the kernel on init regardless of whether nvme_fabrics was loaded as a module or compiled in, making it a reliable indicator that NVMe-oF TCP is ready to use.
Related issues
Fixes: #6158
Checklist:
guidelines in the developer
guide.
Request
notes
updated with breaking and/or notable changes for the next major release.
Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:
/retest ci/centos/<job-name>: retest the<job-name>after unrelatedfailure (please report the failure too!)