Repository navigation
Conversation
yummybomb
added this pull request to stack #522
October 16, 2026 15:24
In mixed-size mode the driver drops a profile from every VF's creatable_vgpu_types once its GPU holds as many vGPUs as that profile's equal-size instance limit, even while a placement remains. Remember the profiles each GPU has advertised, and when none advertises the requested one, let the driver decide by configuring a clean VF per supporting GPU under the allocation lock.
Merge the dirty-VF repair and hidden-profile probe into one loop so a VF reselected after the driver refuses a GPU is reset like any other dirty VF, instead of being claimed while it may still hold open VFIO handles. When the driver rejects a hidden profile, reset unclaimed dirty siblings on that GPU and retry once, since one of them may hold the last placement. Only a rejected current_vgpu_type write, now reported as devices.ErrVGPUTypeRejected, moves placement to the next GPU; any other configure error fails the claim. Remember profiles listed after the dirty-VF reset retry, and only offer a hidden profile on a VF whose own creatable list was read and is non-empty.
Advertised profiles were configured after claimVGPU released the lock, so a later claim could take the placement a saved claim had picked, and its configure then failed with no fallback. Write the type for every claim inside the claim loop, so a driver rejection moves to the next GPU whether or not the profile is advertised. Reset the VF if saving the claim fails, since it is now configured before the claim exists.
…le check When every GPU that advertised a profile refused its write, the selector still counted their creatable lists and reported "no available VF". Count only VFs still in the candidate set, so the claim fails with the not-creatable error as it does for hidden profiles. Count driver rejections per GPU in hypeman_instances_vgpu_driver_rejections_total, return an error instead of saving an unconfigured claim if the selected VF is somehow not a candidate, and note that the later configure call in create and start only re-checks quarantine and the type.
The driver puts each new vGPU at the lowest free placement for its type, which on an L40S lands half of the 2Q vGPUs on placements a 4Q needs. After configuring a type, move the vGPU to the free placement that blocks the fewest free placements of larger profiles, read from nvidia-smi and cached per GPU. Ties and any failure keep the driver's placement.
yummybomb
force-pushed
the
hypeship/vgpu-placement-packing
branch
from
October 16, 2026 21:59
d934a3c to
f87d76f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #511. Review that first; this PR only adds the last commit.
Summary
In mixed-size (heterogeneous) mode, each vGPU occupies a fixed run of its GPU's placement slots, and each profile may only start at the driver's heterogeneous placement IDs. The driver puts a new vGPU at the lowest free placement for its type. On an L40S that scatters 2Q across the runs 4Q needs:
Change
configurewrites and verifiescurrent_vgpu_type, and before any VM attaches,packPlacementmoves the vGPU by writingplacement_id.current_vgpu_typeandplacement_idof the GPU's other configured VFs.nvidia-smi vgpu -s -v -i <GPU>andnvidia-smi -q -i <GPU>, read the same way as the infra role's heterogeneous-mode check.nvidia-smidoesn't run it on every claim.nvidia-smifailing, an unreadable sibling or one with an unknown type, a refusedplacement_idwrite, or a read-back mismatch each log and keep the driver's placement.claimVGPUunder the allocation lock, so siblings can't change underneath it.Validation
nvidia-smi vgpu -s -voutput from a dev L40S, in testdata):configure: writesplacement_idand loads the table once per GPU. It keeps the driver's placement in equal-size mode, whennvidia-smifails, when a sibling's type is unknown, and when the driver refuses the placement write.configure, keep-on-tie, the mode check, the unknown-sibling check, the cache, counting only free larger placements, excluding the VF itself, and only choosing free targets.go test ./lib/devices/andgo test -race ./lib/instances/ -run 'VGPU|VendorVFIO'pass.lib/devicesandlib/instancesbuild for darwin.devices.ConfigureVGPUandDestroyVGPUagainst an empty card, with no VMs attached:current_vgpu_type.Notes