studio: show system-wide VRAM in the multi-GPU System tab view on ROCm - #7216
Conversation
The System tab's per-GPU list comes from get_visible_gpu_utilization. When amd-smi is unavailable (always on Windows, minimal Linux installs) it fell back to torch, whose readings are process-local: on Windows WDDM hands each process its own budget, so a model held by the separate llama-server process read as ~0 VRAM used even with the GPU full (unslothai#7072). The primary-GPU endpoint already compensates with system-wide sources -- Windows Performance Counters (Task Manager's source) and Linux DRM sysfs -- but the multi-device endpoint never got those fallbacks. Add per-GPU variants of both sources and overlay them onto the torch fallback: _rocm_windows_perf_counter_vram_per_adapter_gb() attributes Dedicated Usage per physical adapter (phys_<N> in the counter instance name), and _rocm_linux_sysfs_vram_per_card_gb() reads mem_info_vram_{used,total} per DRM card. _overlay_system_wide_vram() applies them to the device list, ROCm-only, best-effort: unmatched adapters and ambiguous card counts keep the torch figures, and NVIDIA paths are untouched. Fixes unslothai#7072
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f38b9e1386
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…nblock the loop Five review fixes on the multi-GPU system-wide VRAM overlay: 1. Linux: match DRM cards to devices by PHYSICAL index instead of a positional zip, so a reordering visibility mask (HIP_VISIBLE_DEVICES=1,0) no longer swaps each card's figures onto the other GPU (which would mislead auto_select_gpu_ids and the coexistence checks). An index with no matching card keeps its torch figures. 2. Linux: skip the overlay for a device whose sysfs total is below torch's -- on unified-memory APUs (Strix Halo) mem_info_vram_total is only the small dedicated slice while torch sees the GTT-backed pool, and _apply_unified_memory_correction already defines larger-total-wins. 3. Windows: group counter instances by adapter LUID, not the phys_<N> suffix -- separate adapters each read phys_0, which collapsed every GPU into key 0. LUIDs are mapped to 0-based positions by ascending value as the closest stand-in for device order. 4. Windows: pair the system-wide usage with the physical capacity from get_device_properties (as the primary-GPU fallback does) -- under WDDM mem_get_info's "total" is the process budget, which misreported capacity and pushed utilization to 100%. 5. Run get_visible_gpu_utilization off the event loop in the /hardware/visible route (asyncio.to_thread, the repo's convention): the ROCm fallbacks can shell out to PowerShell with a 5s timeout, which would stall every other request while the System view polls.
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2006754c2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The overlay matches its per-GPU sources (Windows perf counters, Linux sysfs) by physical device index, but under a UUID/MIG visibility mask the torch fallback enumerates ordinals and reports index_kind == "relative", where `index` is a visible ordinal, not a physical id. Applying the overlay there let card/adapter 0's system-wide VRAM overwrite the torch reading of a process that actually exposes physical GPU 1, misleading auto_select_gpu_ids and the coexistence checks. Gate the overlay on index_kind == "physical"; relative-index paths keep the torch fallback.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1dbed7733
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The multi-GPU system-wide VRAM overlay is now Linux-only. The Windows per-adapter Performance Counter path could not be made correct: the wildcard Get-Counter query also returns non-ROCm/iGPU adapters and LUID order is not the ROCm device order, so an adapter's usage could be overlaid onto the wrong GPU; and it read only Dedicated Usage, missing WDDM shared memory on unified-memory GPUs (Strix Halo), overstating free VRAM. Rather than misattribute VRAM and skew placement decisions, Windows keeps the process-local torch fallback (no regression vs before this PR); Linux DRM sysfs -- matched by physical index -- still fixes unslothai#7072 for the reporter's native-Linux ROCm case. Removes _rocm_windows_perf_counter_vram_per_adapter_gb and _torch_props_total_gb.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8b976360de
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
… cards
_rocm_linux_sysfs_vram_per_card_gb dropped cards with a zero total or unreadable
files and then the overlay enumerated the compacted list, so if card0 was
dropped, card1's usage was assigned to physical GPU index 0 (equal-capacity GPUs
slip past the unified-memory total guard). Return {card_number: (used, total)}
and match a device to its card number directly: a hole stays a hole -- device 0
keeps its torch figures when card0 is absent, and card1 maps to device 1.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8dcd5edd87
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
When a non-amdgpu adapter (Intel iGPU, a display-only card) owns an earlier DRM slot, DRM card numbers stop equalling ROCm device ordinals -- Intel card0 plus AMD card1/card2 gives ROCm devices 0/1, so keying the sysfs overlay by card number handed ROCm device 1 card1's data (AMD device 0) and left device 0 on stale torch figures, corrupting free-VRAM placement on equal-capacity GPUs. Only amdgpu cards expose mem_info_vram_*, so the glob already excludes foreign adapters; order the surviving cards by their PCI address (ROCm/HIP's default device order, read from each card's device symlink) and key by that position -- the ROCm physical ordinal, which is what the overlay matches against dev index. An unreadable / zero-total amdgpu card still consumes its ordinal so a later card is never renumbered onto its slot.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9090500bbc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
ROCR_VISIBLE_DEVICES filters physical GPUs at the HSA/ROCr layer, and a HIP_VISIBLE_DEVICES set on top selects WITHIN that already-filtered set (apply_gpu_ids sets HIP while leaving an inherited ROCR mask in place). When both are active _get_parent_visible_gpu_spec() prefers the HIP value, so the reported device index is a ROCR-relative ordinal, not a physical GPU id -- overlaying DRM-sysfs figures by that index would pull another GPU's usage (e.g. ROCR=2,3 + HIP=1 is physical GPU 3, but the overlay would read card 1), and equal-capacity cards bypass the total-size safeguard. Detect layered masks and keep torch's process-local figures there rather than risk misattribution; a single mask still leaves the index physical and is overlaid as before.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46615b6f71
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The overlay guard only skipped the case where sysfs total < torch total (unified-memory APUs), so a partitioned ROCm device (MI300 in CPX mode) -- where HIP exposes several logical devices per physical card but sysfs reports the whole card's aggregate -- passed the guard: the card total exceeds a partition's torch total, and the overlay overwrote the partition with whole-card usage and capacity, letting downstream selection think a partition had the entire card free. Require the sysfs card total to match the torch device total (within ~10%) so a mismatch in either direction -- unified memory (sysfs smaller) or partitioning (sysfs larger) -- keeps torch's figures.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c6512d0fc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Two remaining mismatches between the reported device index and the DRM card the overlay reads: - On ROCm the HIP layer honors CUDA_VISIBLE_DEVICES as well as HIP_VISIBLE_DEVICES, so a CUDA mask composed over ROCR layers identically: ROCR=2,3 with CUDA=1 is physical GPU 3, yet the spec reports the ROCR value [2,3] and the device was labeled index 2, overlaying card 2's usage onto GPU 3. The layered check now treats ROCR combined with either HIP or CUDA as layered. - The ROCm device set is now enumerated by bound driver (device/driver resolves to amdgpu) instead of by the presence of mem_info_vram_*. An AMD device with incomplete sysfs support (some APUs expose no VRAM files at all) was omitted by the glob entirely and shifted every later card down one ordinal, letting a similar-capacity GPU pass the total guard with another device's usage. Such a card now consumes its ordinal and simply yields no entry.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e3069bbba
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Two remaining ways the reported device index could be matched to the wrong DRM card: - GPU_DEVICE_ORDINAL is a supported ROCm visibility variable that _get_parent_visible_gpu_spec() never consults, so GPU_DEVICE_ORDINAL=1 surfaces physical GPU 1 as torch ordinal 0 and it was mislabeled index 0, overlaying card 0's usage onto GPU 1. The mask check now covers it, and is renamed _rocm_device_index_unreliable() to say what it actually decides. - driver == amdgpu is only a SUPERSET of the ROCm-visible set: an amdgpu-bound adapter HIP cannot enumerate (an unsupported older AMD GPU beside a supported one) still took an ordinal and shifted every real compute device. There is no torch-side PCI identity to match against, so the overlay now requires the amdgpu card count to equal the device count -- exactly the condition under which position-in-PCI-order is a sound 1:1 mapping. Any disagreement keeps torch's process-local figures: less informative, never misattributed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e775dfb4e0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The card-count guard compared the amdgpu card list against the VISIBLE device list, so any visibility mask disabled the overlay outright: HIP_VISIBLE_DEVICES=1,3 on a four-GPU host gives two devices against four cards. Those masked GPUs then kept reporting process-local torch usage, hiding VRAM held by llama-server and letting the training/chat placement checks overestimate free memory -- the exact problem the overlay exists to fix. The count check now applies only when no visibility mask is active, which is the case where the reported devices really are the whole host and a mismatch means an amdgpu adapter ROCm cannot enumerate is shifting the ordinals. Under a mask the subset is expected, so each device's physical index is validated individually instead: the per-card lookup bounds-checks it and the total-size guard rejects a card whose capacity does not match the device's.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bca1a5ab0b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Every mapping bug on this PR came from the same root cause: there was no authoritative link between a reported device index and a DRM card, so the overlay kept inferring one positionally and each heuristic broke on a new host shape -- foreign adapters on earlier DRM slots, cards with no VRAM sysfs, and most recently amdgpu-bound adapters HIP cannot enumerate, which the count guard could only catch on an unmasked host and therefore missed under any mask. Use the link ROCm itself enumerates from. KFD topology (/sys/class/kfd/kfd/topology/nodes/<N>/properties) lists exactly the GPUs HIP exposes -- GPU nodes in node-id order are HIP's device order -- and each carries its PCI location, so index N there IS physical device N with a stable identity. DRM sysfs now supplies system-wide VRAM keyed by that same PCI address, and the overlay is a join on it. Every previous skew becomes a failed join rather than a misattribution: an unenumerable adapter has no KFD node so it never takes an ordinal, a foreign adapter contributes no entry, and a masked subset resolves each physical index directly. That removes the count heuristic and its mask exception entirely. With no KFD topology there is no identity to join on, so the overlay is skipped rather than guessing positionally.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 048fd3f73a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Three ways the identity map could still be built on a false premise: - The NVIDIA open kernel module registers KFD topology nodes with a positive SIMD count, so an earlier NVIDIA node shifted every AMD ordinal and ROCm device 1 resolved to AMD GPU 0. GPU nodes now require vendor_id 4098 (0x1002), the same filter install.sh already applies for this exact reason. - A GPU node with an unreadable properties file or no location_id was skipped, which silently shifted every later ordinal. Both now fail the whole map closed, so the overlay is disabled rather than misattributing. - A container exposing only some render devices through device cgroups sets no visibility variable, yet torch compacts what it can see to ordinals from zero while the host-mounted KFD and DRM trees still list every GPU. Nothing in the reported payload distinguishes that from a full host, and torch exposes no PCI id to check against, so the overlay now runs only when host visibility is positively verified: no visibility mask AND device count equal to the host GPU count. That also subsumes the previous layered-mask and GPU_DEVICE_ORDINAL checks, so _rocm_device_index_unreliable() is gone. This trades coverage for correctness: masked subsets and filtered containers now keep torch's process-local figures instead of a mapping that cannot be verified.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a290eab63
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| pci_by_ordinal = _rocm_kfd_gpu_pci_ids() | ||
| if not pci_by_ordinal: | ||
| return | ||
| if _rocm_visibility_mask_active() or len(devices) != len(pci_by_ordinal): |
There was a problem hiding this comment.
When a launcher exports an identity/all-GPU mask such as HIP_VISIBLE_DEVICES=0,1 or CUDA_VISIBLE_DEVICES=0,1 on a two-GPU ROCm host, the reported indices are still the host physical ordinals and the KFD count matches, but this predicate returns before applying the sysfs overlay. In that common managed-launcher case the System tab and free-VRAM placement logic fall back to process-local torch values, so VRAM held by the separate llama-server can be missed; this is distinct from the earlier filtered-subset concern because the mask does not actually filter or reorder the devices.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Intended. An identity mask like HIP_VISIBLE_DEVICES=0,1 does preserve order, but the overlay cannot verify from torch alone that the index is a host-physical ordinal, so it conservatively keeps torch's figures under any mask. Correctness over completeness: it never attributes another card's usage. The unmasked path, which is the common Studio case, still gets the system-wide overlay.
Resolve studio/backend/utils/hardware/hardware.py: keep the merged Windows ROCm per-adapter VRAM path (unslothai#7238) and add this branch's Linux KFD/DRM overlay alongside it. The old _rocm_windows_perf_counter_vram_gb helper was removed on main; drop its stub here.
The docstring claimed a reordering mask keeps each card on the right GPU, but the overlay skips any active visibility mask and keeps torch's figures. State the actual gating instead.
|
Merged latest main and resolved the conflict in I also tightened the Both suites pass locally: |
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Collapse the verbose docstrings and inline explanations added for the Linux ROCm system-wide VRAM overlay to succinct one-liners, keeping the non-obvious rationale (fail-closed KFD mapping, PCI-identity join, mask gating, the 10% whole-card guard). Comments only, no behavior change.
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
3 participants
The System tab's per-GPU list comes from get_visible_gpu_utilization. When
amd-smi is unavailable (always on Windows, minimal Linux installs) it fell back
to torch, whose readings are process-local: on Windows WDDM hands each process
its own budget, so a model held by the separate llama-server process read as
~0 VRAM used even with the GPU full (#7072). The primary-GPU endpoint already
compensates with system-wide sources -- Windows Performance Counters (Task
Manager's source) and Linux DRM sysfs -- but the multi-device endpoint never
got those fallbacks.
Add per-GPU variants of both sources and overlay them onto the torch fallback:
rocm_windows_perf_counter_vram_per_adapter_gb() attributes Dedicated Usage per
physical adapter (phys in the counter instance name), and
rocm_linux_sysfs_vram_per_card_gb() reads mem_info_vram{used,total} per DRM
card. _overlay_system_wide_vram() applies them to the device list, ROCm-only,
best-effort: unmatched adapters and ambiguous card counts keep the torch
figures, and NVIDIA paths are untouched.
Fixes #7072