[yummybomb]

Problem

In mixed-size vGPU mode, the driver drops a profile from creatable_vgpu_types once a GPU holds as many vGPUs as that profile's equal-size limit (12 for L40S-4Q), even when a placement for it is still free. Hypeman trusts those lists, so 4Q creates fail on busy cards and /resources is wrong in both directions. On one busy production day, 36% of 4Q claim failures had a free placement the driver had stopped listing.

Will release cold start gpus after this PR.

Fix

NVML's creatable-placements query (what nvidia-smi vgpu -c -v shows) stays accurate in that state (L40S, driver 580.178.05):

vGPUs on card	free 4Q placements	sysfs lists 4Q	NVML lists 4Q
11	2	yes	yes
12	2	no	yes
13	1	no	yes
14	0	no	no

• Placement and /resources read capacity from NVML. libnvidia-ml.so.1 is loaded at runtime with purego, so no cgo is needed. A query takes about 20 ms.
• Placement queries NVML on every claim. /resources and the resource metrics read cached placements instead: the cache loads at startup, refreshes in the background after every vendor VFIO configure or reset, and refreshes again on every one-minute reconcile pass. Free and quarantined VFs are read fresh on each request, so a claim or release counts at once; the placements it frees or takes catch up when the background refresh lands. Refreshes coalesce and run one at a time, so an older snapshot never replaces a newer one.
• A free VF can take any profile its GPU has a free placement for. GPU selection is unchanged.
• claimVGPU writes the type under the allocation lock, before saving the claim, so the next claim sees the placement as taken. If configure or save fails, the VF is reset. A restart between the write and the save, or a failed claim whose reset also failed, leaves a configured VF with no claim; the reconciler resets those on every pass, so a refused reset is retried, and counts refusals as hypeman_instances_vgpu_reconcile_failures_total{stage="reset_unclaimed_vfs"}.
• /resources available is, summed over GPUs, the smaller of free placements and free healthy VFs. Counts aren't additive across profiles; the OpenAPI description says so.
• Admission only needs free-VF counts, so it no longer queries NVML.

Fallback

A GPU that NVML fails for, or that isn't in mixed-size mode, uses the sysfs lists as on main; the other GPUs keep NVML. If NVML can't load or times out (5 s), every GPU falls back. Each fallback increments hypeman_resources_gpu_nvml_fallbacks_total{reason}, and a warning is logged when a fallback starts or its error changes (plus one info line when NVML recovers); a query whose only failures are GPUs in equal-size mode (drift after a driver reload or GPU reset) counts as equal_size, not error. NVML queries run one at a time, and waiting for the slot counts against the same 5 s, so callers queued behind a hung query never enter the driver. A timed-out call is abandoned and later queries skip NVML until it returns, so a hung driver costs one timeout, not one per claim.

Testing

• Unit tests cover placement, /resources counts, per-GPU and whole-host fallback, timeout and stuck handling, claim ordering, and resets. go test ./lib/devices ./lib/resources and go test ./lib/instances -run 'VGPU|VendorVFIO' pass, also with -race.
• On a dev L40S host (driver 580.178.05), running e837c75 merged onto main, with one GPU at 12 vGPUs so sysfs hid 4Q:
  • /resources matched NVML's free placements at every step (4Q 7, 2Q 14; main reported 26 and 45).
  • 4Q creates landed on placements sysfs hid; main placed them on the other GPU.
  • 5 concurrent 4Q creates with 3 free placements: 3 succeeded, 2 failed fast, no VF left configured. Stop/start re-claimed.
  • To force a per-GPU NVML failure, an empty GPU was temporarily put in equal-size mode: only it fell back (4Q 36, vs 57 if the whole host fell back), and NVML picked it back up once mixed-size mode was restored.
• The equal-size fallback reason, the periodic unclaimed VF reset, the /resources placement cache (in-memory reads, coalesced refreshes, refresh after configure and reset) and the one-at-a-time NVML slot have unit tests, all mutation-checked; none of them has run on a GPU host.
• The NVML binding has no CI coverage.

Notes

• Concurrent GPU creates on one host wait on each other's ~20 ms query: in a burst of 20 on the dev host, the NVML part of claiming took up to about 0.4 s for the last create.
• GPUs are still ranked by framebuffer load, not free placements.
• CGO_ENABLED=0 builds (goreleaser) become dynamically linked against glibc. Fleet builds already are.
• hypeman_resources_gpu_profile_slots drops on busy hosts after deploy (it now counts placements, not VFs). Anything using it as a free-VF count should read hypeman_resources_gpu_slots instead, which is unchanged.
• Report vGPU capacity exhaustion as insufficient resources #510 sees hidden placements without changes and adds one more query per create.
• hypeman keeps NVML initialized, which holds /dev/nvidia0..N, /dev/nvidiactl and /dev/nvidia-caps/* open (checked on the dev host), the same nodes DCGM holds. The SR-IOV cycle in GPU.md that fixes a wedged VF now stops hypeman along with DCGM and persistenced; stopping hypeman leaves running VMs alone. The driver upgrade runbook already stops hypeman. The fds are close-on-exec, so VMMs don't inherit them, and switching a GPU's vGPU mode still works while hypeman holds them.

---

Note

High Risk
Changes core vGPU allocation, reconciliation, and capacity reporting on production GPU hosts; NVML holds device nodes and mis-fallback or stuck-query behavior could block or mis-advertise placements until recovery.

Overview
Vendor VFIO vGPU placement and /resources profile counts now use NVML creatable placements (via runtime libnvidia-ml.so.1 + purego) instead of per-VF creatable_vgpu_types, which under-report busy mixed-size GPUs. Claims still query NVML on each allocation; /resources and metrics read a cached placement snapshot refreshed after configure/reset and on reconcile, with sysfs fallback, 5s timeouts, serialized queries, and hypeman_resources_gpu_nvml_fallbacks_total.

Claim lifecycle on vendor VFIO writes current_vgpu_type under the allocation lock before persisting metadata, resets the VF if configure or save fails, and the reconciler resets unclaimed dirty VFs every pass (new reset_unclaimed_vfs failure stage). Profile available is redefined as min(free placements, free VFs) per GPU, non-additive across profiles.

Admission and lightweight GPU status skip profile/NVML work (GetGPUStatus withProfiles flag). Docs and OpenAPI describe the new semantics; SR-IOV recovery runbooks note stopping hypeman because NVML holds GPU device nodes.

^{Reviewed by Cursor Bugbot for commit 9fe16b3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 2c9c09c. Configure here.}

[sjmiller609]

WDYT? open to push back on this. But the main concern is that "hypeman resources" output would not be accurate. I would be best if we know up front how many of any given profile can fit, not just discovering what doesn't fit after trying.

potential issues with this approach:

• hypeman resources still reports inaccurate remaining slots per profile.
• allocation discovers capacity through rejected writes rather than shared capacity accounting.

potential solutions:

• share capacity accounting between allocation and hypeman resources.
• calculate remaining slots from usable VRAM minus configured reservations, accounting for placement constraints and free, healthy VFs.
• keep driver-rejection handling as a fallback.

[github-actions]

-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at 7702279 (pushed)

go get github.com/kernel/hypeman-go-staging@7702279d2470abb7b6e57fe4a283037fbd21d8c2

✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

232 files generated at 28db799 (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at deee22c (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note

Level	Code	Message	Targets

Build metadata

Build	bd_76bFg2Bp-mirthful-dust
Timestamp	2026-10-21T15:38:14.235Z
stlc	8413509
Spec hash	0f6faa2c6d93
Config hash	659c3687c3f0

---

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-10-21 15:38:40 UTC

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github.com/​ebitengine/​purego@​v0.11.2

View full report

[sjmiller609]

• cache GPU capacity for /resources; update allocation records on successful configure/release, refresh profile counts asynchronously after changes, and periodically reconcile with NVML.
  • follow lib/instances/admission_allocations.go: startup hydration, lifecycle-updated allocation records, periodic reconciliation, and in-memory reads. profile counts need placement-aware recalculation, not simple increments/decrements.
• bound concurrent NVML calls before starting them; the current timeout guard allows many requests to enter a hung driver simultaneously.
• retry orphaned VF cleanup periodically; a temporary startup failure can leave capacity stranded and prevent new allocations (not sure - wdyt?)

[yummybomb]

all three addressed

• agree on the orphaned VF cleanup: it now runs on every vGPU reconcile pass (at startup, then every minute), so a reset that fails once, e.g. while a VFIO handle is still open, gets retried a minute later instead of waiting for the allocator to pick that VF, which it may never do if placement skips the host. refusals are counted under hypeman_instances_vgpu_reconcile_failures_total{stage="reset_unclaimed_vfs"}. it still takes the allocation lock, so it can't race a claim, and reset still refuses a VF with an open VFIO handle.

• /resources reads cached placements now (refreshed after each configure/reset and every reconcile pass), and NVML queries take a single slot before starting, so nothing queues into a hung driver.

[sjmiller609]

Thank you!