Skip to content

clean up a vm's iso storage repos when it fails to start on xenserver - #14425

Open
Damans227 wants to merge 1 commit into
apache:mainfrom
Damans227:xenserver-remove-iso-sr-on-failed-start
Open

Damans227 wants to merge 1 commit into
apache:mainfrom
Damans227:xenserver-remove-iso-sr-on-failed-start

Conversation

@Damans227

@Damans227 Damans227 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #11984

When a vm fails to start on a XenServer or XCP-ng host, the cleanup removes the vm, its disks and network cards, but leaves its iso storage repositories (<vm>-ISO and <vm>-CONFIGDRIVE-ISO) behind. The stop that follows finds no vm and skips them, and the next start attempt creates another one with the same name. After a few retries there are several repositories with that name, and from then on even a normal stop cant remove them, because the lookup gives up when it finds more than one. This removes both repositories in the start failure cleanup, the same way a normal stop does.

Test:

Added a unit test that fails a vm start and checks both iso repositories of that vm are removed.

On a lab with one XCP-ng 8.2 host: an L2 network with Config Drive for userdata, cluster memory overcommit set to 4, and a 12 GB offering, so CloudStack places the vm but the host fails the start with HOST_NOT_ENOUGH_FREE_MEMORY. Deploy a vm with userdata and an iso attached, start it three times, then change it to a small offering, start it and stop it. After each step, list the vm's storage repos on the host with xe sr-list.

Before: every failed start leaves one more CONFIGDRIVE-ISO repo behind, and the normal stop at the end cant remove them either.

== start 1
== storage repos for i-2-135-VM on the host:
count: 2
== start 2
== storage repos for i-2-135-VM on the host:
count: 4
== start 3
== storage repos for i-2-135-VM on the host:
count: 6
== start with a small offering, then stop
== storage repos for i-2-135-VM on the host after the stop:
uuid ( RO)          : 36637543-7bc3-e47b-c1bb-c983781dad96
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
uuid ( RO)          : 21d92447-2b4e-0742-4ac2-371fee2811a2
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
uuid ( RO)          : 7d2818ca-466c-5e8d-0f12-48b201f3e255
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
uuid ( RO)          : ffbb2a75-d81f-4ded-08d0-d50b26b3c0a2
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
uuid ( RO)          : fb81ef54-0655-5818-aeb3-3aa84671fbec
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
uuid ( RO)          : eb655195-c797-dac5-5da6-3c0bbec0f1a2
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
uuid ( RO)          : 666af859-bba2-4ce1-f787-a9136d19eb9e
    name-label ( RW): i-2-135-VM-CONFIGDRIVE-ISO
count: 7

After: nothing is left after any failed start or after the stop.

== start 1
errortext": "Unable to orchestrate the start of VM instance {\
== storage repos for i-2-137-VM on the host:
count: 0
== start 2
errortext": "Unable to orchestrate the start of VM instance {\
== storage repos for i-2-137-VM on the host:
count: 0
== start 3
errortext": "Unable to orchestrate the start of VM instance {\
== storage repos for i-2-137-VM on the host:
count: 0
== start with a small offering, then stop
"state": "Running"
"state": "Stopped"
== storage repos for i-2-137-VM on the host after the stop:
count: 0

Host log (/var/log/xensource.log) for one failed start, the repo is removed right after the vm:

audit] SR.set_name_label: SR = 'dcd7b1d3-a9e1-ce34-6350-bab18d5ca44a (...)' name-label = 'i-2-137-VM-CONFIGDRIVE-ISO'
audit] VM.start_on: VM = 'f1e47c62-6978-af14-9648-cd33e0007848 (i-2-137-VM)'
Server_error(HOST_NOT_ENOUGH_FREE_MEMORY
audit] VM.destroy: VM = 'f1e47c62-6978-af14-9648-cd33e0007848 (i-2-137-VM)'
audit] SR.forget: SR = 'dcd7b1d3-a9e1-ce34-6350-bab18d5ca44a (i-2-137-VM-CONFIGDRIVE-ISO)'

@Damans227

Copy link
Copy Markdown
Collaborator Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@Damans227 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 20.39%. Comparing base (d3f0700) to head (527fbd6).

Additional details and impacted files
@@            Coverage Diff            @@
##               main   #14425   +/-   ##
=========================================
  Coverage     20.39%   20.39%           
+ Complexity    21283    21281    -2     
=========================================
  Files          6481     6481           
  Lines        588040   588042    +2     
  Branches      72021    72021           
=========================================
+ Hits         119957   119960    +3     
+ Misses       455052   455043    -9     
- Partials      13031    13039    +8     
Flag Coverage Δ
uitests 3.70% <ø> (ø)
unittests 21.70% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19587

@Damans227
Damans227 requested a review from vishesh92 October 16, 2026 17:57
@Damans227
Damans227 marked this pull request as ready for review October 16, 2026 18:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Configdrive/storage repository issue on XCP-ng

2 participants