diff --git a/private-cicd/benchmark/ontap/AGENTS.md b/private-cicd/benchmark/ontap/AGENTS.md index b79e7f29c9f6..40ea169fdbb6 100644 --- a/private-cicd/benchmark/ontap/AGENTS.md +++ b/private-cicd/benchmark/ontap/AGENTS.md @@ -23,7 +23,7 @@ Scale matrix: [ONTAP Plugin - CloudStack Operations, Scale & Parallel Test Matri Test ids written to the CSVs and report map to the page sections as follows: | Test id | Page section | What it measures | -|---|---|---| +| --- | --- | --- | | `pool_sequential_create` | 5.1.1 | Sequential create (1 pool at a time, cumulative to N) | | `pool_sequential_delete` | 5.1.2 | Sequential delete (1 pool at a time, from N remaining) | | `pool_concurrent_create` | 6.1.1 | Parallel pool creation | diff --git a/private-cicd/benchmark/ontap/USAGE.md b/private-cicd/benchmark/ontap/USAGE.md index 9082a095a49f..cad587532aac 100644 --- a/private-cicd/benchmark/ontap/USAGE.md +++ b/private-cicd/benchmark/ontap/USAGE.md @@ -26,7 +26,7 @@ this file is deliberately just a lookup table / cheat sheet. ## 1. Prerequisites | # | Requirement | Notes | -|---|---|---| +| --- | --- | --- | | 1 | Python 3.9+ | No other version-specific features used; anything 3.9-3.12 should work. | | 2 | Network access to a CloudStack management server's API (`/client/api`) | Admin credentials with permission to create/delete storage pools, service/disk offerings, and deploy/destroy VMs. | | 3 | Network access (HTTPS/443) from **your machine** to the ONTAP cluster's management LIF | Only needed if you run the storage-pool scripts' `--cleanup-only` helper functions that talk to ONTAP directly for orphan checks; the benchmark scripts themselves only ever call the CloudStack API, never ONTAP directly. | @@ -70,7 +70,7 @@ cluster IDs, and ONTAP connection details (see inline comments in ## 2. Shared library modules (not run directly) | File | Purpose | -|---|---| +| --- | --- | | `benchmark_support.py` | Shared run ids, resource names, `mean_seconds`, and the one log format used by every script. | | `cloudstack_client.py` | Minimal CloudStack HTTP/REST client (session-key login, generic `call()`, automatic `queryAsyncJobResult` polling). Used by every benchmark script. | | `storage_pool_common.py` | Shared helpers for the storage-pool scripts: `createStoragePool`/`deleteStoragePool` request building, CSV logging (`RawLogger`, `append_summary_csv`), cleanup-by-filter, config loading. | @@ -82,7 +82,7 @@ Reproduces Confluence sections 5.1 (sequential) and 6.1 (concurrency) - `createStoragePool` / `deleteStoragePool` timing at scale. | Script | What it does | Typical command | -|---|---|---| +| --- | --- | --- | | `run.py` | Entry point. Loads `config.yaml` and runs the test named by `--test`. | `python3 run.py --test storage-pool-sequential --config config.yaml` | | `benchmark_storage_pool_sequential.py` | Creates storage pools one at a time (per protocol) up to N=30, then deletes them one at a time, logging every call and reporting checkpoint totals at N = 1, 5, 10, 20, 30. | `python3 run.py --test storage-pool-sequential --config config.yaml` | | `benchmark_storage_pool_concurrency.py` | For each concurrency level C (default 2, 5, 10, 20, 30), creates C pools in parallel via a thread pool, records wall-clock time for the whole batch, then deletes the same C pools in parallel. | `python3 run.py --test storage-pool-concurrency --config config.yaml` | @@ -90,7 +90,7 @@ Reproduces Confluence sections 5.1 (sequential) and 6.1 (concurrency) - Common flags (both scripts): | Flag | Meaning | -|---|---| +| --- | --- | | `--config PATH` | Config YAML to use (default `config.yaml`). Protocols, checkpoints, and concurrency levels are read from here. | | `--run-id ID` | Reuse a run id. Auto-generated otherwise as `RUN__<4 hex chars>`. Letters, digits, and underscores only — a hyphen is rewritten to `_` because storage-pool names (and the ONTAP volume behind them) disallow `-`. Use the **same id** across sequential and concurrency to merge one report. | | `--dry-run` | Simulate timings with no real API calls - use this first to sanity-check your config. | @@ -100,7 +100,7 @@ Common flags (both scripts): `run.py` also requires `--test`. Protocols, checkpoints, and concurrency levels are not flags; change them in `config.yaml`. | Flag | Meaning | -|---|---| +| --- | --- | | `--test NAME` | Required. `storage-pool-sequential`, `storage-pool-concurrency`, `vm-instance-sequential`, `vm-instance-concurrency`, or `vm-instance-combined`. | ## 4. VM-instance benchmark scripts @@ -126,7 +126,7 @@ VM getting both a root disk and a data disk on the same tagged pool. ### Scripts | Script | What it does | Typical command | -|---|---|---| +| --- | --- | --- | | `benchmark_vm_instance_sequential.py` | Deploys VMs one at a time (per protocol) up to N=30, then destroys them one at a time, reporting checkpoint totals at N = 1, 2, 5, 10, 20, 30. | `python3 run.py --test vm-instance-sequential --config config.yaml` | | `benchmark_vm_instance_concurrency.py` | For each concurrency level C, deploys C VMs in parallel, records wall-clock time, then destroys the same C VMs in parallel. | `python3 run.py --test vm-instance-concurrency --config config.yaml` | | `benchmark_vm_instance_combined.py` | Runs the sequential matrix immediately followed by the concurrency matrix, under one shared run id, in a single invocation. Use once you already trust your config/environment. | `python3 run.py --test vm-instance-combined --config config.yaml` | @@ -143,7 +143,7 @@ with `--cleanup-only` once done. ## 5. Report rendering | Script | What it does | Typical command | -|---|---|---| +| --- | --- | --- | | `render_report.py` | Turns a run's `raw_ops_.csv` + `summary_.csv` into markdown tables (section layout plus Min/Avg/P95/Max rows for the Results Log). Writes to stdout and to `results/report_.md`. | `python3 render_report.py --run-id RUN_20260927_081500_a1b2 --cloudstack-build 4.23.0.0-SNAPSHOT --ontap-version 9.17.1` | For VM-instance runs (which write `raw_ops_vm_.csv` / @@ -156,7 +156,7 @@ python3 render_report.py --run-id RUN_20260927_081500_a1b2 \ ``` | Flag | Meaning | -|---|---| +| --- | --- | | `--run-id ID` | **Required.** The run id used by the benchmark script(s). | | `--output-dir DIR` | Where the CSVs live / the report gets written (default `results`). | | `--cloudstack-build STR` | Label only, stamped into the Results Log rows. | @@ -169,7 +169,7 @@ python3 render_report.py --run-id RUN_20260927_081500_a1b2 \ ## 6. Config files at a glance | File | Committed? | Purpose | -|---|---|---| +| --- | --- | --- | | `config.example.yaml` | Yes | Sanitized template - copy to `config.yaml` and fill in your lab's real values. | | `config.yaml` | **No** (gitignored) | Your real, lab-specific config (real IPs/credentials) - never commit this. | | `config_sanity.yaml` | **No** (gitignored) | Optional scratch config some contributors keep locally for quick one-off dry-runs; not part of the checked-in tooling. | @@ -180,7 +180,7 @@ All written under `results/` (gitignored - these are run artifacts, not source): | Pattern | Produced by | Contents | -|---|---|---| +| --- | --- | --- | | `raw_ops_.csv` | storage-pool scripts | One row per individual API call (timestamps, duration, success/failure, error). | | `raw_ops_vm_.csv` | VM-instance scripts | Same, for VM deploy/destroy calls. | | `summary_.csv` / `summary_vm_.csv` | both | One row per checkpoint/concurrency level (totals, averages, success/failure counts). | diff --git a/test/integration/plugins/ontap/README.md b/test/integration/plugins/ontap/README.md index ec787d003461..6a40a1402c2e 100644 --- a/test/integration/plugins/ontap/README.md +++ b/test/integration/plugins/ontap/README.md @@ -43,7 +43,10 @@ for the deployment, health-gate, and artifact contracts. test/integration/plugins/ontap/ ├── ontap.cfg # Environment config (IPs, credentials, zone info) ├── ontap_test_base.py # Shared base class and ONTAP REST client -├── TEST_CASES.md # Full test case reference table (119 tests) +├── guest_ssh.py # Guest SSH (paramiko) for the VM workflow suites +├── setup_env.sh # venv: Marvin plus requirements.txt +├── requirements.txt # setuptools, pyvmomi, paramiko pins +├── TEST_CASES.md # Full test case reference table (151 tests) ├── README.md # This file │ ├── nfs3/ @@ -54,7 +57,7 @@ test/integration/plugins/ontap/ │ ├── volume/ │ │ └── test_volume_lifecycle.py # Volume create/delete/negative-delete │ ├── instance/ -│ │ └── test_vm_volume_attach.py # Pool + volume + VM + attach/resize/detach + template cache +│ │ └── test_vm_volume_attach.py # Pool + volume + VM + resize, snapshots, attach/detach + template cache │ └── template/ │ └── test_template_cache_negative.py # Template-cache boundary cases │ @@ -66,7 +69,7 @@ test/integration/plugins/ontap/ ├── volume/ │ └── test_volume_lifecycle.py # LUN create/delete/negative-delete ├── instance/ - │ └── test_vm_volume_attach.py # Pool + LUN + VM + attach/resize/LUN-map lifecycle + template cache + │ └── test_vm_volume_attach.py # Pool + LUN + VM + resize, snapshots, LUN-map lifecycle + template cache └── template/ └── test_template_cache_negative.py # Template-cache boundary cases ``` @@ -122,6 +125,38 @@ python3 -m pip install --user \ python3 -c "import marvin; print('Marvin OK')" ``` +The VM workflow suites also need a venv with pinned dependencies. From the repo root: + +```bash +bash test/integration/plugins/ontap/setup_env.sh +``` + +The script is safe to re-run. It creates `test/integration/plugins/ontap/.venv` when that directory is missing, then installs into it: + +- Marvin (from `MARVIN_ARCHIVE` when set, otherwise from `tools/marvin` after generating `commands.xml`) +- the pins in `requirements.txt`: `setuptools>=40.3.0,<81`, `pyvmomi==8.0.2.0.1`, and `paramiko>=3.4,<4` + +`run_tests.sh` uses that venv automatically. + +#### Paramiko (guest SSH) + +`nfs3/instance/test_vm_volume_attach.py` and `iscsi/instance/test_vm_volume_attach.py` log into the guest through `guest_ssh.py`. The lab template is CentOS 5.5, and its sshd only offers SHA-1 key exchange (`diffie-hellman-group14-sha1`) and `ssh-rsa` host keys. Paramiko 4.x removed SHA-1, so those logins fail unless the venv has Paramiko 3.x. + +`setup_env.sh` installs the pin from `requirements.txt`: + +```bash +test/integration/plugins/ontap/.venv/bin/pip install -r test/integration/plugins/ontap/requirements.txt +``` + +`guest_ssh.check_paramiko()` checks for `diffie-hellman-group14-sha1` before the first guest connection and tells you to re-run `setup_env.sh` when the active interpreter has Paramiko 4. Guest login itself comes from `ontap.cfg` (`cloudstack.templateUsername` / `cloudstack.templatePassword`). + +Confirm the venv before a VM workflow run: + +```bash +test/integration/plugins/ontap/.venv/bin/python -c \ + "import paramiko; print(paramiko.__version__); assert 'diffie-hellman-group14-sha1' in paramiko.Transport._kex_info" +``` + --- ## Configuration — `ontap.cfg` @@ -347,13 +382,13 @@ another ONTAP pool is present. | NFS3 Pool with Volumes | `nfs3/pool/test_pool_with_volumes.py` | 12 | Lifecycle with a live volume, deletion with a pre-deleted FlexVol or export policy, and host cases for an inactive libvirt pool, a read-only NFS mount, and a deleted mount point | | NFS3 Zone-Scoped Pool | `nfs3/pool/test_zone_scoped_pool.py` | 5 | Zone-scoped pool create, grow and shrink with CloudStack/ONTAP/KVM verification, disable, enable, and delete | | NFS3 Volume Lifecycle | `nfs3/volume/test_volume_lifecycle.py` | 5 | Volume is metadata-only; FlexVol unchanged on delete | -| NFS3 VM + Volume Attach | `nfs3/instance/test_vm_volume_attach.py` | 13 | Full VM lifecycle with stopped-VM qcow2 grow; ROOT on tagged pool seeds/reuses template cache, which survives VM delete | +| NFS3 VM + Volume Attach | `nfs3/instance/test_vm_volume_attach.py` | 29 | Full VM lifecycle with stopped-VM qcow2 grow, guest SSH, volume and VM snapshots; ROOT on tagged pool seeds/reuses template cache, which survives VM delete | | NFS3 Template Cache Negative | `nfs3/template/test_template_cache_negative.py` | 3 | Tag mismatch; undersized pool; out-of-band cache delete | | iSCSI Pool Lifecycle | `iscsi/pool/test_pool_lifecycle.py` | 21 | Create rejects for aggregate space, invalid capacitybytes, and a duplicate FlexVol name; grow to 300 TiB, reject grow past 300 TiB, safe shrink, disable, resize while disabled, enable, maintenance, grow and shrink in maintenance, delete + igroups, then re-create with a volume (LUN), reject shrink below used capacity, deploy a VM and attach the volume, cleanup, plus empty-pool deletion with a pre-deleted FlexVol or igroups | | iSCSI Pool with Volumes | `iscsi/pool/test_pool_with_volumes.py` | 13 | Lifecycle with a live LUN, deletion with a pre-deleted FlexVol or igroups, maintenance with pre-deleted LUN maps, and host cases for a logged-out iSCSI session, an existing session during volume delete, and a replaced by-path symlink | | iSCSI Zone-Scoped Pool | `iscsi/pool/test_zone_scoped_pool.py` | 5 | Zone-scoped pool create, grow and shrink with CloudStack/ONTAP verification, disable, enable, and delete | | iSCSI Volume Lifecycle | `iscsi/volume/test_volume_lifecycle.py` | 5 | LUN created per CS volume; LUN removed on delete | -| iSCSI VM + Volume Attach | `iscsi/instance/test_vm_volume_attach.py` | 13 | Full VM lifecycle; stopped-VM LUN grow and LUN-map checks; ROOT on tagged pool seeds/reuses `cs_tmpl_*` LUN cache | +| iSCSI VM + Volume Attach | `iscsi/instance/test_vm_volume_attach.py` | 29 | Full VM lifecycle; stopped-VM LUN grow, LUN-map checks, guest SSH, volume and VM snapshots; ROOT on tagged pool seeds/reuses `cs_tmpl_*` LUN cache | | iSCSI Template Cache Negative | `iscsi/template/test_template_cache_negative.py` | 3 | Tag mismatch; undersized pool; out-of-band cache delete | For the goal, dependencies, and exact success criteria of every individual test, see [TEST_CASES.md](TEST_CASES.md). @@ -364,12 +399,13 @@ For the goal, dependencies, and exact success criteria of every individual test, | Symptom | Likely cause | Fix | | --------- | ------------- | ----- | +| `paramiko … has no SHA-1 key exchange` / guest SSH fails on CentOS 5.5 | Paramiko 4.x is installed; the guest sshd only offers SHA-1 | Re-run `bash test/integration/plugins/ontap/setup_env.sh` and run tests with `test/integration/plugins/ontap/.venv` (`paramiko>=3.4,<4`) | | `ModuleNotFoundError: No module named 'ontap_test_base'` | Missing `PYTHONPATH` prefix | Prefix every run with `PYTHONPATH=test/integration/plugins/ontap` | | `Marvin Init Failed` | CloudStack API unreachable | Check `mgtSvrIp:8096` is reachable; restart `cloudstack-management` | | `Lost connection to MySQL` | MySQL not accepting remote connections | Enable remote MySQL access (see Prerequisites §3) | | `sh: python: command not found` (repeated) | Marvin internal call — harmless on macOS | Ignore; Marvin Init still succeeds | | Pool state never reaches `Maintenance` | KVM agent not responding | Check `cloudstack-agent` on KVM host; verify host is connected in CloudStack UI | -| iSCSI `test_09` error 530 | KVM guest does not ACK SCSI hot-unplug | Known environment limitation — see TEST_CASES.md Suite 10 note | +| iSCSI `test_24` error 530 | KVM guest does not ACK SCSI hot-unplug | Detach of the data LUN from a running VM; see TEST_CASES.md Suite 10 `test_24_detach_volume_from_vm` | | ONTAP REST `401 Unauthorized` | Wrong credentials in `ontap.cfg` | Verify `username`/`password` under `ontap` section | | `ISCSI protocol is not enabled on SVM` / many pool tests EXCEPTION then FAIL | SVM iSCSI service off or no `data_iscsi` LIF | Enable iSCSI on the SVM and assign `default-data-iscsi` (or `default-data-blocks`) to at least one data LIF; `run_tests.sh` now fails this check before Marvin starts | | `No ready KVM user template available` | Template still downloading | Re-run `setup_zone` (step 12 waits for template readiness); or wait in CloudStack UI | diff --git a/test/integration/plugins/ontap/TEST_CASES.md b/test/integration/plugins/ontap/TEST_CASES.md index 0b235925c0f0..4145e3c5eb79 100644 --- a/test/integration/plugins/ontap/TEST_CASES.md +++ b/test/integration/plugins/ontap/TEST_CASES.md @@ -19,7 +19,7 @@ # ONTAP Integration Test Cases -Complete reference for all 119 test cases across 12 test suites. +Complete reference for all 151 test cases across 12 test suites. Each suite is sequential — tests must run in numbered order; each step builds on state created by the previous step. --- @@ -133,7 +133,7 @@ Each suite is sequential — tests must run in numbered order; each step builds **File:** `nfs3/instance/test_vm_volume_attach.py` **Class:** `TestOntapVMVolumeAttach` **Tag:** `vm_volume_workflow` -**Total:** 13 tests | **Scope:** end-to-end — NFS3 pool, data volume resize, running VM (ROOT on the ONTAP pool via a tagged compute offering), primary template cache seed / reuse / survive VM delete, attach/detach lifecycle +**Total:** 29 tests | **Scope:** end-to-end — NFS3 pool, data volume resize, running VM (ROOT on the ONTAP pool via a tagged compute offering), primary template cache seed / reuse / survive VM delete, attach/detach, guest SSH, volume snapshots, and VM snapshots. Guest login uses `guest_ssh.py` (paramiko) against the CentOS 5.5 template. | # | Test method | Goal | Depends on | CloudStack success criteria | ONTAP success criteria | Type | | --- | ------------- | ------ | ------------ | ----------------------------- | ------------------------ | ------ | @@ -147,9 +147,25 @@ Each suite is sequential — tests must run in numbered order; each step builds | 06 | `test_06_grow_stopped_vm_volume` | Grow the attached volume while the VM is stopped | test_05 | Volume is `Ready` at the requested byte size | `qemu-img info` reports the requested virtual size; ONTAP file remains present | positive | | 07 | `test_07_shrink_rejected` | Attempt to shrink the grown qcow2 volume with `shrinkok=true` | test_06 | Resize is rejected; grown size remains unchanged | qcow2 virtual size remains unchanged | negative | | 08 | `test_08_start_vm_volume_accessible` | Start the stopped VM | test_07 | `vm.state == "Running"`; KVM data-disk capacity equals target | FlexVol still `online` | positive | -| 09 | `test_09_detach_volume_from_vm` | Hot-detach the ONTAP volume from the running VM (TDS Detach NFS3) | test_08 (`vm`, `volume`) | `volume.virtualmachineid` cleared; `volume.state == "Ready"` | FlexVol still `online`; data file **still present** (NFS3: file persists until `deleteVolume`, not on detach) | positive | -| 10 | `test_10_grow_detached_volume` | Grow the detached, materialized data volume | test_09 | API and DB size equal target | qcow2 virtual size equals target | positive | -| 11 | `test_11_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, force-delete pool | test_10 | VM no longer listed; spool_ref still Ready after VM expunge; volume no longer listed; pool no longer listed | Cache file present after VM expunge; FlexVol deleted; export policy deleted | cleanup | +| 09 | `test_09_open_ssh_and_write_root_data_files` | Port-forward SSH to the guest and write ROOT and DATA marker files | test_08 (`vm`, `volume`, guest network) | ROOT and DATA both on the ONTAP pool; DATA still attached; SSH login succeeds; both markers read back | FlexVol still `online` | positive | +| 10 | `test_10_root_volume_snapshot_create` | VS-01: snapshot the ROOT volume of the running VM | test_09 | Snapshot `BackedUp`; VM stays `Running` | One new FlexVol snapshot | positive | +| 11 | `test_11_data_volume_snapshot_create` | VS-03: snapshot the attached DATA volume (kept for test_14) | test_10 | Snapshot `BackedUp`; VM stays `Running` | One new FlexVol snapshot, kept until test_14 | positive | +| 12 | `test_12_volume_revert_running_vm_rejected` | VS-19: revert a volume snapshot while the VM is Running | test_11 | Revert is rejected | FlexVol unchanged | negative | +| 13 | `test_13_root_volume_revert` | VS-11: write A, snapshot ROOT, write B, stop, revert, start | test_12 | After revert and start, ROOT marker is A | New FlexVol snapshot for the snapshot; revert snapshot removed in tearDown | positive | +| 14 | `test_14_data_volume_revert` | Snapshot DATA again, revert to the test_11 snapshot then to this one, then delete both while the VM is running | test_11, test_13 | Older revert restores the test_11 DATA marker and the later snapshot stays listed; later revert restores its marker; both snapshots are then gone and the VM stays `Running` | Both FlexVol snapshots remain through the older revert, then both are removed | positive | +| 15 | `test_15_vm_snapshot_create_running` | VMS-01: VM snapshot of the running VM (kept for test_17) | test_14 | VM snapshot `Ready`; VM stays `Running` | One new FlexVol snapshot | positive | +| 16 | `test_16_vm_snapshot_create_stopped` | VMS-03: stop, VM snapshot, start (kept for test_17) | test_15 | VM snapshot `Ready` | One new FlexVol snapshot | positive | +| 17 | `test_17_vm_snapshot_s1_running_s2_stopped_revert` | VMS-09: revert to the test_15 snapshot, then to the test_16 snapshot, then delete both while the VM is running | test_15, test_16 | Each revert finishes; guest file matches that snapshot; both VM snapshots are then gone and the VM stays `Running` | FlexVol snapshots survive the reverts, then both are removed | positive | +| 18 | `test_18_vm_snapshot_duplicate_requests` | VMS-53: two VM snapshot requests for the same VM at once | test_17 | Both snapshots `Ready` (a rejected request is retried) | Two new FlexVol snapshots | positive | +| 19 | `test_19_deploy_second_vm` | Deploy VM-B whose ROOT disk is on the same ONTAP pool | test_18 | VM-B `Running`; ROOT `storageid` = pool | ROOT file present on the shared FlexVol | positive | +| 20 | `test_20_volume_revert_leaves_other_vm_untouched` | VS-28: revert VM-A's ROOT | test_19 | VM-A's ROOT marker is restored; VM-B's file is unchanged | FlexVol stays `online` | positive | +| 21 | `test_21_concurrent_vm_snapshots_two_vms` | VMS-54: VM snapshots of both VMs at the same time | test_20 | Both VM snapshots `Ready` | Two new FlexVol snapshots | positive | +| 22 | `test_22_concurrent_volume_snapshot_and_revert` | Volume snapshot of VM-A's ROOT while VM-B's ROOT reverts | test_21 | Snapshot `BackedUp` and revert both finish (retried on error 530) | New FlexVol snapshot for the volume snapshot | positive | +| 23 | `test_23_concurrent_vm_snapshot_and_revert` | VM snapshot of VM-A while VM-B reverts to its VM snapshot | test_22 | Both operations finish; VM-A snapshot `Ready` | New FlexVol snapshot for VM-A | positive | +| 24 | `test_24_detach_volume_from_vm` | Hot-detach the ONTAP volume from the running VM (TDS Detach NFS3) | test_23 (`vm`, `volume`) | `volume.virtualmachineid` cleared; `volume.state == "Ready"` | FlexVol still `online`; data file **still present** (NFS3: file persists until `deleteVolume`, not on detach) | positive | +| 25 | `test_25_grow_detached_volume` | Grow the detached, materialized data volume | test_24 | API and DB size equal target | qcow2 virtual size equals target | positive | +| 26 | `test_26_destroy_vm_with_vm_snapshot` | VMS-38: expunge VM-B while it still has a VM snapshot | test_19 (VM-B) | VM-B no longer listed | VM-B's FlexVol snapshot removed | positive | +| 27 | `test_27_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, force-delete pool | test_25, test_26 | VM no longer listed; spool_ref still Ready after VM expunge; volume no longer listed; pool no longer listed | Cache file present after VM expunge; FlexVol deleted; export policy deleted | cleanup | --- @@ -250,7 +266,7 @@ Each suite is sequential — tests must run in numbered order; each step builds **File:** `iscsi/instance/test_vm_volume_attach.py` **Class:** `TestOntapVMVolumeAttachISCSI` **Tag:** `iscsi_vm_workflow` -**Total:** 13 tests | **Scope:** end-to-end — iSCSI pool, data volume resize (LUN), running VM (ROOT on the ONTAP pool via a tagged compute offering), `cs_tmpl_` LUN cache seed / reuse / survive VM delete, attach/stop/start/detach lifecycle +**Total:** 29 tests | **Scope:** end-to-end — iSCSI pool, data volume resize (LUN), running VM (ROOT on the ONTAP pool via a tagged compute offering), `cs_tmpl_` LUN cache seed / reuse / survive VM delete, attach/stop/start/detach, guest SSH, volume snapshots, and VM snapshots. LUN-map checks are limited to the data volume's LUN. Guest login uses `guest_ssh.py` (paramiko) against the CentOS 5.5 template. | # | Test method | Goal | Depends on | CloudStack success criteria | ONTAP success criteria | Type | | --- | ------------- | ------ | ------------ | ----------------------------- | ------------------------ | ------ | @@ -263,12 +279,28 @@ Each suite is sequential — tests must run in numbered order; each step builds | 05 | `test_05_stop_vm_lun_unmapped` | Stop VM — LUN-maps must be removed (TDS VM Stop iSCSI) | test_04 | `vm.state == "Stopped"` | 0 LUN-maps; LUN itself **still present** in FlexVol | positive | | 06 | `test_06_grow_stopped_vm_volume` | Grow the attached volume while the VM is stopped | test_05 | Volume is `Ready` at the requested byte size | LUN `space.size` equals the requested size; LUN remains unmapped | positive | | 07 | `test_07_shrink_rejected` | Attempt to shrink the grown LUN with `shrinkok=true` | test_06 | Resize is rejected; grown size remains unchanged | LUN `space.size` remains unchanged | negative | -| 08 | `test_08_start_vm_lun_remapped` | Start VM — LUN-maps must be re-created (TDS VM Start iSCSI) | test_07 | `vm.state == "Running"`; KVM data-disk capacity equals target | ≥1 LUN-map re-created | positive | -| 09 | `test_09_detach_volume_from_vm` | Hot-detach the iSCSI volume from the running VM (TDS Detach iSCSI) | test_08 (`vm`, `volume`) | `volume.virtualmachineid` cleared | 0 LUN-maps; LUN still in FlexVol | positive ⚠️ | -| 10 | `test_10_grow_detached_volume` | Grow the detached, materialized data volume | test_09 | API and DB size equal target | LUN `space.size` equals target | positive | -| 11 | `test_11_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, force-delete pool | test_10 | VM gone; spool_ref still Ready after VM expunge; volume gone; pool gone | `cs_tmpl_*` present after VM expunge; FlexVol deleted; all LUNs and igroups deleted | cleanup | - -> ⚠️ **test_09 known status:** iSCSI hot-detach from a running VM relies on the KVM guest acknowledging the SCSI device removal. On this environment the guest does not acknowledge in time, causing CloudStack error 530. This is a KVM-host-level or guest-template limitation, not a test code defect. +| 08 | `test_08_start_vm_lun_remapped` | Start VM — LUN-maps must be re-created (TDS VM Start iSCSI) | test_07 | `vm.state == "Running"` | ≥1 LUN-map re-created | positive | +| 09 | `test_09_open_ssh_and_write_root_data_files` | Port-forward SSH to the guest and write ROOT and DATA marker files | test_08 (`vm`, `volume`, guest network) | ROOT and DATA both on the ONTAP pool; DATA still attached; SSH login succeeds; both markers read back | Data LUN still mapped | positive | +| 10 | `test_10_root_volume_snapshot_create` | VS-01: snapshot the ROOT volume of the running VM | test_09 | Snapshot `BackedUp`; VM stays `Running` | One new FlexVol snapshot | positive | +| 11 | `test_11_data_volume_snapshot_create` | VS-03: snapshot the attached DATA volume (kept for test_14) | test_10 | Snapshot `BackedUp`; VM stays `Running` | One new FlexVol snapshot, kept until test_14 | positive | +| 12 | `test_12_volume_revert_running_vm_rejected` | VS-20: revert a volume snapshot while the VM is Running | test_11 | Revert is rejected | FlexVol unchanged | negative | +| 13 | `test_13_root_volume_revert` | VS-12: write A, snapshot ROOT, write B, stop, revert, start | test_12 | After revert and start, ROOT marker is A | New FlexVol snapshot for the snapshot; revert snapshot removed in tearDown | positive | +| 14 | `test_14_data_volume_revert` | Snapshot DATA again, revert to the test_11 snapshot then to this one, then delete both while the VM is running | test_11, test_13 | Older revert restores the test_11 DATA marker and the later snapshot stays listed; later revert restores its marker; both snapshots are then gone and the VM stays `Running` | Both FlexVol snapshots remain through the older revert, then both are removed | positive | +| 15 | `test_15_vm_snapshot_create_running` | VM snapshot of the running VM, ROOT + DATA LUNs (kept for test_17) | test_14 | VM snapshot `Ready`; VM stays `Running` | One new FlexVol snapshot | positive | +| 16 | `test_16_vm_snapshot_create_stopped` | Stop, VM snapshot, start (kept for test_17) | test_15 | VM snapshot `Ready` | One new FlexVol snapshot | positive | +| 17 | `test_17_vm_snapshot_s1_running_s2_stopped_revert` | VMS-09: revert to the test_15 snapshot, then to the test_16 snapshot, then delete both while the VM is running | test_15, test_16 | Each revert finishes; guest file matches that snapshot; both VM snapshots are then gone and the VM stays `Running` | FlexVol snapshots survive the reverts, then both are removed | positive | +| 18 | `test_18_vm_snapshot_duplicate_requests` | VMS-53: two VM snapshot requests for the same VM at once | test_17 | Both snapshots `Ready` (a rejected request is retried) | Two new FlexVol snapshots | positive | +| 19 | `test_19_deploy_second_vm` | Deploy VM-B whose ROOT LUN is on the same ONTAP pool | test_18 | VM-B `Running`; ROOT on pool | One more non-cache LUN; `cs_tmpl_*` still one | positive | +| 20 | `test_20_volume_revert_leaves_other_vm_untouched` | VS-28: revert VM-A's ROOT | test_19 | VM-A's ROOT marker is restored; VM-B's file is unchanged | FlexVol stays `online`; both ROOT LUNs remain | positive | +| 21 | `test_21_concurrent_vm_snapshots_two_vms` | VMS-55: VM snapshots of both VMs at the same time | test_20 | Both VM snapshots `Ready` | Two new FlexVol snapshots | positive | +| 22 | `test_22_concurrent_volume_snapshot_and_revert` | Volume snapshot of VM-A's ROOT while VM-B's ROOT reverts | test_21 | Snapshot `BackedUp` and revert both finish (retried on error 530) | New FlexVol snapshot for the volume snapshot | positive | +| 23 | `test_23_concurrent_vm_snapshot_and_revert` | VM snapshot of VM-A while VM-B reverts to its VM snapshot | test_22 | Both operations finish; VM-A snapshot `Ready` | New FlexVol snapshot for VM-A | positive | +| 24 | `test_24_detach_volume_from_vm` | Hot-detach the iSCSI volume from the running VM (TDS Detach iSCSI) | test_23 (`vm`, `volume`) | `volume.virtualmachineid` cleared | 0 LUN-maps for the data LUN; LUN still in FlexVol | positive ⚠️ | +| 25 | `test_25_grow_detached_volume` | Grow the detached, materialized data volume | test_24 | API and DB size equal target | LUN `space.size` equals target | positive | +| 26 | `test_26_destroy_vm_with_vm_snapshot` | Expunge VM-B while it still has a VM snapshot | test_19 (VM-B) | VM-B no longer listed | VM-B ROOT LUN gone; `cs_tmpl_*` still present | positive | +| 27 | `test_27_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, delete pool | test_25, test_26 | VM gone; spool_ref still Ready after VM expunge; volume gone; pool gone | `cs_tmpl_*` present after VM expunge; FlexVol deleted; all LUNs and igroups deleted | cleanup | + +> ⚠️ **test_24 known status:** iSCSI hot-detach from a running VM relies on the KVM guest acknowledging the SCSI device removal. On this environment the guest does not acknowledge in time, causing CloudStack error 530. This is a KVM-host-level or guest-template limitation, not a test code defect. --- @@ -310,12 +342,12 @@ Each suite is sequential — tests must run in numbered order; each step builds | NFS3 Pool with Volumes | NFS3 | Cluster | 12 | ✅ | | NFS3 Zone-Scoped Pool | NFS3 | Zone | 5 | ✅ | | NFS3 Volume Lifecycle | NFS3 | Cluster | 5 | ✅ | -| NFS3 VM + Volume Attach | NFS3 | Cluster | 13 | ⏳ resize cases pending lab validation | +| NFS3 VM + Volume Attach | NFS3 | Cluster | 29 | ⏳ resize and snapshot cases pending lab validation | | NFS3 Template Cache Negative | NFS3 | Cluster | 3 | 🆕 | | iSCSI Pool Lifecycle | iSCSI | Cluster | 21 | ✅ | | iSCSI Pool with Volumes | iSCSI | Cluster | 13 | ✅ | | iSCSI Zone-Scoped Pool | iSCSI | Zone | 5 | ✅ | | iSCSI Volume Lifecycle | iSCSI | Cluster | 5 | ✅ | -| iSCSI VM + Volume Attach | iSCSI | Cluster | 13 | ⏳ resize cases pending lab validation; detach limitation | +| iSCSI VM + Volume Attach | iSCSI | Cluster | 29 | ⏳ resize and snapshot cases pending lab validation; detach limitation | | iSCSI Template Cache Negative | iSCSI | Cluster | 3 | 🆕 | -| **Total** | | | **119** | **VM resize cases pending lab validation** | +| **Total** | | | **151** | **Resize and snapshot cases pending lab validation** | diff --git a/test/integration/plugins/ontap/guest_ssh.py b/test/integration/plugins/ontap/guest_ssh.py new file mode 100644 index 000000000000..749e918214e8 --- /dev/null +++ b/test/integration/plugins/ontap/guest_ssh.py @@ -0,0 +1,216 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +SSH access to the guest VMs deployed by the ONTAP VM workflow tests +(``*/instance/test_vm_volume_attach.py``). + +The template is CentOS 5.5; its login comes from ontap.cfg +(``cloudstack.templateUsername`` / ``cloudstack.templatePassword``, default +root/password). Guests sit on an isolated network, so each VM gets a +port-forwarding rule (public port -> 22) on the network's source-NAT IP. +Markers are plain files written and read with paramiko. +""" + +import logging +import random +import time +import uuid + +import paramiko + +from marvin.cloudstackAPI import ( + createPortForwardingRule as createPortForwardingRuleAPI, + deletePortForwardingRule as deletePortForwardingRuleAPI, + listPublicIpAddresses as listPublicIpAddressesAPI, +) + +logger = logging.getLogger("OntapGuestSsh") + +ROOT_MARKER = "/root/ontap-marker" +DATA_MOUNT = "/mnt/data" +DATA_MARKER = DATA_MOUNT + "/ontap-marker" + +_DATA_MOUNT_CMD = ( + "mkdir -p {m}; " + "mountpoint -q {m} 2>/dev/null || grep -q ' {m} ' /proc/mounts || " + "mount /dev/vdb {m} 2>/dev/null || mount /dev/sdb {m} 2>/dev/null" +).format(m=DATA_MOUNT) + +_DATA_FORMAT_CMD = ( + "mkdir -p {m}; " + "grep -q ' {m} ' /proc/mounts || " + "mount /dev/vdb {m} 2>/dev/null || mount /dev/sdb {m} 2>/dev/null || " + "{{ if [ -b /dev/vdb ]; then d=/dev/vdb; else d=/dev/sdb; fi; " + "mkfs.ext3 -F -q $d && mount $d {m}; }}" +).format(m=DATA_MOUNT) + + +def check_paramiko(): + """Fail fast when paramiko cannot talk to the CentOS 5.5 guest sshd.""" + if "diffie-hellman-group14-sha1" not in paramiko.Transport._kex_info: + raise GuestSshError( + "paramiko %s has no SHA-1 key exchange, which the CentOS 5.5 guest " + "requires. Run test/integration/plugins/ontap/setup_env.sh and use " + "its .venv (it pins paramiko<4)." % paramiko.__version__ + ) + + +def guest_credentials(config): + """Return (username, password) for the template from ontap.cfg.""" + cs_cfg = config.get("cloudstack", {}) + return (cs_cfg.get("templateUsername", "root"), + cs_cfg.get("templatePassword", "password")) + + +def new_token(label="snapshot-test"): + """Return readable, unique text to write into a snapshot test file.""" + readable_label = "".join( + char if char.isalnum() or char in ("-", "_") else "-" + for char in label + ).strip("-") + return "%s-%s" % (readable_label or "snapshot-test", + uuid.uuid4().hex[:12]) + + +class GuestSshError(Exception): + """SSH to the guest or a guest command failed.""" + + +class GuestSsh(object): + """Run shell commands on a guest as root over SSH.""" + + def __init__(self, host, port=22, username="root", password="password", + connect_timeout=600, retry_interval=10): + self.host = host + self.port = int(port) + self.username = username + self.password = password + self.connect_timeout = connect_timeout + self.retry_interval = retry_interval + self._ssh = None + + def __enter__(self): + self.connect() + return self + + def __exit__(self, exc_type, exc, tb): + self.close() + return False + + def connect(self): + """Connect, retrying until the guest's sshd is up after boot.""" + check_paramiko() + deadline = time.time() + self.connect_timeout + last_error = None + while time.time() < deadline: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + ssh.connect(self.host, port=self.port, username=self.username, + password=self.password, timeout=15, + allow_agent=False, look_for_keys=False) + self._ssh = ssh + logger.info("SSH connected to %s:%d", self.host, self.port) + return + except Exception as exc: + last_error = exc + ssh.close() + time.sleep(self.retry_interval) + raise GuestSshError("SSH to %s:%d failed within %ds: %s" + % (self.host, self.port, self.connect_timeout, last_error)) + + def close(self): + if self._ssh is not None: + self._ssh.close() + self._ssh = None + + def run(self, command, check=True, timeout=300): + """Run a command; return stdout. Raise on non-zero exit if check.""" + if self._ssh is None: + self.connect() + _, stdout, stderr = self._ssh.exec_command(command, timeout=timeout) + status = stdout.channel.recv_exit_status() + out = stdout.read().decode("utf-8", "replace") + err = stderr.read().decode("utf-8", "replace") + if check and status != 0: + raise GuestSshError("'%s' on %s exited %d: %s" + % (command, self.host, status, err.strip())) + return out + + def write_root_marker(self, token): + self.run("echo '%s' > %s && sync" % (token, ROOT_MARKER)) + + def read_root_marker(self): + return self.run("cat %s 2>/dev/null; true" % ROOT_MARKER).strip() + + def write_data_marker(self, token): + self.run(_DATA_FORMAT_CMD) + self.run("echo '%s' > %s && sync" % (token, DATA_MARKER)) + + def read_data_marker(self): + self.run(_DATA_MOUNT_CMD, check=False) + return self.run("cat %s 2>/dev/null; true" % DATA_MARKER).strip() + + +# --------------------------------------------------------------------------- +# Port forwarding +# --------------------------------------------------------------------------- + +def source_nat_ip(apiclient, network_id): + """Return the source-NAT public IP record for an isolated network.""" + cmd = listPublicIpAddressesAPI.listPublicIpAddressesCmd() + cmd.associatednetworkid = network_id + cmd.issourcenat = True + cmd.listall = True + ips = apiclient.listPublicIpAddresses(cmd) or [] + if not ips: + raise GuestSshError("No source-NAT IP found for network %s" % network_id) + return ips[0] + + +def open_ssh_port(apiclient, vm_id, network_id, public_port=None): + """Forward a public port on the source-NAT IP to the VM's port 22. + + Returns (public_ip, public_port, rule_id). + """ + ip = source_nat_ip(apiclient, network_id) + last_error = None + for _ in range(10): + port = public_port or random.randint(20000, 29999) + cmd = createPortForwardingRuleAPI.createPortForwardingRuleCmd() + cmd.ipaddressid = ip.id + cmd.privateport = 22 + cmd.publicport = port + cmd.protocol = "TCP" + cmd.virtualmachineid = vm_id + cmd.openfirewall = True + try: + rule = apiclient.createPortForwardingRule(cmd) + return ip.ipaddress, port, rule.id + except Exception as exc: + last_error = exc + if public_port: + break + raise GuestSshError("Could not create SSH port forward for VM %s: %s" + % (vm_id, last_error)) + + +def close_ssh_port(apiclient, rule_id): + cmd = deletePortForwardingRuleAPI.deletePortForwardingRuleCmd() + cmd.id = rule_id + apiclient.deletePortForwardingRule(cmd) diff --git a/test/integration/plugins/ontap/iscsi/instance/test_vm_volume_attach.py b/test/integration/plugins/ontap/iscsi/instance/test_vm_volume_attach.py index 49dde8d1828e..ab46b96e6a3b 100644 --- a/test/integration/plugins/ontap/iscsi/instance/test_vm_volume_attach.py +++ b/test/integration/plugins/ontap/iscsi/instance/test_vm_volume_attach.py @@ -34,20 +34,20 @@ - startVirtualMachine → LUN-map is re-created - detachVolume → LUN-map is removed -Tests are numbered test_01 ... test_11, plus test_03a and test_03b, and must +Tests are numbered test_01 ... test_27, plus test_03a and test_03b, and must run in that order. Each step builds on the shared state established by the previous step. The VM is deployed with a compute offering whose storage tags match the pool, so its ROOT volume lands on the ONTAP pool and exercises the primary template cache (``cs_tmpl_`` LUN: seed / reuse / survive VM delete) -alongside the data-volume resize workflow. LUN-map assertions are filtered to -the data volume's LUN, so the ROOT and cache LUNs in the same FlexVol do not -affect them. +alongside the data-volume resize and snapshot workflow. LUN-map assertions are +filtered to the data volume's LUN, so the ROOT and cache LUNs in the same +FlexVol do not affect them. Workflow: 01 Create iSCSI primary storage pool on ONTAP (tagged -tmpl-cache) - 02 Create a CloudStack data volume on the iSCSI pool (LUN on ONTAP) + 02 Create and grow a detached CloudStack data volume (LUN on ONTAP) 03 Deploy a VM — ROOT on ONTAP; seeds the cs_tmpl_* cache LUN 03a Deploy a second VM — reuses the cache LUN (one more ROOT clone LUN only) 03b Expunge the second VM — its ROOT LUN is removed; cache LUN survives @@ -56,9 +56,37 @@ 06 Grow the attached volume and ONTAP LUN while the VM is stopped 07 Verify shrinking the ONTAP LUN is rejected without changing its size 08 Start VM — LUN-map is re-created on ONTAP - 09 Detach data volume from running VM — LUN-map removed - 10 Grow the detached data volume - 11 Destroy VM (cache LUN still survives), delete data volume, delete pool + 09 Open an SSH port forward to the VM; write ROOT and DATA files + 10 Snapshot the ROOT volume of the running VM + 11 Snapshot the DATA volume of the running VM (kept for step 14) + 12 Revert a volume snapshot while the VM is Running — rejected + 13 ROOT volume revert: write before-snapshot, snapshot, write + after-snapshot, revert, file is back to before-snapshot + 14 DATA: snapshot again, revert to the step-11 snapshot then to this + one, then delete both while the VM is still running + 15 Write before-s1; VM snapshot of the running VM (kept for step 17) + 16 Write before-s2, stop, VM snapshot (kept for step 17) + 17 Revert to the step-15 snapshot, then to the step-16 snapshot, then + delete both while the VM is still running + 18 Two VM snapshot requests for the same VM at once — both succeed + 19 Deploy a second VM whose ROOT LUN is on the same pool + 20 Revert VM-A's ROOT; VM-B's file is unchanged + 21 VM snapshots of both VMs at the same time + 22 Volume snapshot of VM-A while VM-B's ROOT reverts + 23 VM snapshot of VM-A while VM-B reverts to its VM snapshot + 24 Detach data volume from running VM — LUN-map removed + 25 Grow the detached data volume + 26 Expunge the second VM while it still has a VM snapshot + 27 Destroy VM (cache LUN still survives), delete data volume, delete pool + +Snapshot steps only create and verify; tearDown deletes every snapshot the +step created and waits for its FlexVol snapshot to go. Step 11 keeps its +DATA snapshot so step 14 can revert to it and then delete both DATA +snapshots while the VM is running. Steps 15 and 16 keep their VM snapshots +so step 17 can revert to them and then delete both while the VM is running. + +Guest login for the CentOS 5.5 template comes from ontap.cfg +(cloudstack.templateUsername / templatePassword); see guest_ssh.py. Prerequisites: - CloudStack management server with the NetApp ONTAP plugin deployed @@ -77,6 +105,7 @@ import logging import random import re +import threading import time import unittest @@ -99,9 +128,10 @@ startVirtualMachine as startVirtualMachineAPI, stopVirtualMachine as stopVirtualMachineAPI, ) -from marvin.lib.base import ServiceOffering, StoragePool +from marvin.lib.base import ServiceOffering, Snapshot, StoragePool, VmSnapshot, Volume from marvin.lib.common import list_storage_pools +from guest_ssh import GuestSsh, guest_credentials, new_token, open_ssh_port from ontap_test_base import OntapRestClient, OntapTestBase, get_datacenter_config from helpers import template_cache_util as tcu from helpers.template_cache_base import ( @@ -113,6 +143,8 @@ logger = logging.getLogger("TestOntapVMVolumeAttachISCSI") +TAG = "iscsi_vm_workflow" + # --------------------------------------------------------------------------- # Utility functions @@ -137,6 +169,83 @@ def _wait_for_vm_state(api_client, vm_id, target_state, timeout=300, return None +def _run_concurrent(*callables): + """Start all callables at the same moment; return [(result, exception)].""" + results = [(None, None)] * len(callables) + barrier = threading.Barrier(len(callables)) + + def worker(index, fn): + try: + barrier.wait(timeout=60) + results[index] = (fn(), None) + except Exception as exc: + results[index] = (None, exc) + + threads = [threading.Thread(target=worker, args=(i, fn)) + for i, fn in enumerate(callables)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + return results + + +def _is_rejected(exc): + """True when CloudStack or ONTAP refused the call because another one was in progress.""" + text = str(exc).lower() + return any(token in text for token in ( + "internal error", + "failed to create snapshot", + "failed to create instance snapshot", + "failed to revert", + "another operation", + "other active", + "try again later", + "in progress", + "busy", + )) + + +def _snapshot_and_revert_with_retry(snapshot_fn, revert_fn, attempts=3, pause=15): + """Run a snapshot and a revert together. + + A call that is rejected is retried on its own. A call that already + succeeded is not repeated. Returns the snapshot from snapshot_fn. + """ + snap = None + reverted = False + last_errors = [] + for attempt in range(1, attempts + 1): + jobs = [] + if snap is None: + jobs.append(("snapshot", snapshot_fn)) + if not reverted: + jobs.append(("revert", revert_fn)) + results = _run_concurrent(*(fn for _, fn in jobs)) + last_errors = [] + retryable = True + for (kind, _), (result, exc) in zip(jobs, results): + if exc is None: + if kind == "snapshot": + snap = result + else: + reverted = True + continue + last_errors.append("%s: %s" % (kind, exc)) + if not _is_rejected(exc): + retryable = False + if snap is not None and reverted: + return snap + if not retryable or attempt == attempts: + break + logger.warning("snapshot/revert rejected, retry %d/%d: %s", + attempt, attempts, last_errors) + time.sleep(pause) + raise AssertionError( + "Snapshot-while-revert failed after %d attempt(s): %s" + % (attempt, last_errors)) + + # --------------------------------------------------------------------------- # Test data # --------------------------------------------------------------------------- @@ -156,12 +265,14 @@ class TestData: DETAIL_STORAGE_IP = "storageIP" ONTAP_MIN_VOLUME_SIZE = 1677721600 + # Used when ontap.cfg has neither templateCacheCapacitybytes nor capacitybytes. + DEFAULT_POOL_SIZE = 50 * 1024 ** 3 def __init__(self, storage_ip, svm_name, username, password, scope="CLUSTER", provider="NetApp ONTAP", tags="ontap-iscsi", capacitybytes=None): - if capacitybytes is None: - capacitybytes = TestData.ONTAP_MIN_VOLUME_SIZE * 2 + if not capacitybytes: + capacitybytes = TestData.DEFAULT_POOL_SIZE encoded_password = base64.b64encode(password.encode()).decode() self.testdata = { TestData.ontap: { @@ -220,6 +331,18 @@ class TestOntapVMVolumeAttachISCSI(OntapTestBase): _created_network_id = None # network created by this suite for Advanced zones initial_volume_size = None target_volume_size = None + vm_ssh = None # (public_ip, public_port) forwarded to vm:22 + vm2_ssh = None + vm_snap_running = None # step 15 snapshot, reused by step 17 + vm_snap_stopped = None # step 16 snapshot, reused by step 17 + token_s1 = None + token_s2 = None + kept_ontap_snaps = None + data_snap_older = None # step 11 DATA snapshot, reused by step 14 + data_token_older = None + kept_data_ontap_snaps = None + guest_user = "root" + guest_password = "password" _vol_name_prefix = "OntapISCSIVM" @@ -265,6 +388,8 @@ def setUpClass(cls): cls._setup_cloudstack_resources(config, cls.testdata[TestData.account]) + cls.guest_user, cls.guest_password = guest_credentials(config) + # Discover a ready user KVM template (exclude SYSTEM type) template = find_ready_kvm_template(cls.apiClient, cls.zone.id) cls.template_id = template.id if template is not None else None @@ -282,7 +407,6 @@ def setUpClass(cls): ) cls._cleanup.append(service_offering) cls.service_offering_id = service_offering.id - # Network ID for VM deployment cls.network_id = None zone_type = getattr(cls.zone, "networktype", "Basic") @@ -325,9 +449,11 @@ def setUpClass(cls): def tearDownClass(cls): """ Safety-net cleanup: destroy VMs if still alive, delete the guest - network created for Advanced zones (if not already deleted by test_11), + network created for Advanced zones (if not already deleted by test_27), then delegate pool/volume/account cleanup to the base class. """ + cls._delete_kept_data_snapshot() + cls._delete_kept_vm_snapshots() for vm in (cls.vm2, cls.vm): if vm is None: continue @@ -348,7 +474,7 @@ def tearDownClass(cls): cls.vm = None # Delete the guest network created for this account in Advanced zones. - # test_11 deletes it on the happy path; this is the fallback for + # test_27 deletes it on the happy path; this is the fallback for # mid-suite failures. if cls._created_network_id is not None: try: @@ -461,6 +587,7 @@ def _data_luns(self, volume=None): self.svm_name, self.__class__.pool.name) if lun.get("name") == lun_name] + def _data_lun_size(self): """Return the ONTAP data LUN size in bytes.""" lun_name = self._data_lun_name() @@ -472,6 +599,339 @@ def _data_lun_size(self): size, 0, "ONTAP LUN '%s' did not report space.size" % lun_name) return size + # ---- snapshot step helpers (steps 09-23) --------------------------- + + def setUp(self): + super(TestOntapVMVolumeAttachISCSI, self).setUp() + self._volume_snapshot_ids = [] + self._vm_snapshot_ids = [] + self._extra_volume_ids = [] + self._created_ontap_snaps = set() + + def tearDown(self): + """ + Delete the snapshots and extra volumes a step created, then check + their FlexVol snapshots are gone from ONTAP. + """ + for snap_id in self._vm_snapshot_ids: + try: + VmSnapshot.deleteVMSnapshot(self.apiClient, snap_id) + except Exception as exc: + logger.warning("tearDown: VM snapshot %s: %s" % (snap_id, exc)) + for snap_id in self._volume_snapshot_ids: + try: + Snapshot({"id": snap_id}).delete(self.apiClient) + except Exception as exc: + logger.warning("tearDown: volume snapshot %s: %s" % (snap_id, exc)) + for vol_id in self._extra_volume_ids: + try: + cmd = deleteVolumeAPI.deleteVolumeCmd() + cmd.id = vol_id + self.apiClient.deleteVolume(cmd) + except Exception as exc: + logger.warning("tearDown: volume %s: %s" % (vol_id, exc)) + if self._created_ontap_snaps and not self._wait_ontap_snaps_gone( + self._created_ontap_snaps, fail=False): + logger.warning("tearDown: FlexVol snapshots still present: %s" + % sorted(self._created_ontap_snaps)) + super(TestOntapVMVolumeAttachISCSI, self).tearDown() + + @classmethod + def _destroy_second_vm(cls): + if cls.vm2 is None: + return + try: + cmd = destroyVirtualMachineAPI.destroyVirtualMachineCmd() + cmd.id = cls.vm2.id + cmd.expunge = True + cls.apiClient.destroyVirtualMachine(cmd) + except Exception as exc: + logger.warning("Could not destroy second VM %s: %s" % (cls.vm2.id, exc)) + cls.vm2 = None + cls.vm2_ssh = None + + @classmethod + def _delete_kept_vm_snapshots(cls): + """Delete the step 15/16 snapshots when step 17 did not take them over.""" + for snap in (cls.vm_snap_running, cls.vm_snap_stopped): + if snap is None: + continue + try: + VmSnapshot.deleteVMSnapshot(cls.apiClient, snap.id) + except Exception as exc: + logger.warning("tearDownClass: VM snapshot %s: %s" % (snap.id, exc)) + cls.vm_snap_running = None + cls.vm_snap_stopped = None + cls.token_s1 = None + cls.token_s2 = None + cls.kept_ontap_snaps = set() + + @classmethod + def _delete_kept_data_snapshot(cls): + """Delete the step 11 DATA snapshot when step 14 did not take it over.""" + snap = cls.data_snap_older + if snap is None: + return + try: + Snapshot({"id": snap.id}).delete(cls.apiClient) + except Exception as exc: + logger.warning("tearDownClass: volume snapshot %s: %s" % (snap.id, exc)) + cls.data_snap_older = None + cls.data_token_older = None + cls.kept_data_ontap_snaps = set() + + def _keep_data_snapshot(self, snap, ontap_names, token): + """Leave the step 11 DATA snapshot in place for step 14.""" + if snap.id in self._volume_snapshot_ids: + self._volume_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + self.__class__.data_snap_older = snap + self.__class__.data_token_older = token + self.__class__.kept_data_ontap_snaps = names + + def _adopt_kept_data_snapshot(self): + """Hand the step 11 DATA snapshot to step 14.""" + snap = self.__class__.data_snap_older + token = self.__class__.data_token_older + if snap is None or not token: + self.skipTest("DATA snapshot from test_11 is required") + ontap_names = set(self.__class__.kept_data_ontap_snaps or []) + self._volume_snapshot_ids.append(snap.id) + self._created_ontap_snaps |= ontap_names + self.__class__.data_snap_older = None + self.__class__.data_token_older = None + self.__class__.kept_data_ontap_snaps = set() + return snap, token, ontap_names + + def _delete_volume_snapshot_while_running(self, vm, snap, ontap_names, token): + """Delete a volume snapshot while its VM is still running.""" + Snapshot({"id": snap.id}).delete(self.apiClient) + if snap.id in self._volume_snapshot_ids: + self._volume_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + self._wait_until_gone( + lambda: self._volume_snapshot_listed(snap.id), + "Volume snapshot %s still listed after delete" % snap.id) + self._wait_ontap_snaps_gone(names) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_marker(token, disk="DATA", msg="after volume snapshot delete") + + def _delete_vm_snapshot_while_running(self, vm, snap, ontap_names, token): + """Delete a VM snapshot while that VM is still running.""" + VmSnapshot.deleteVMSnapshot(self.apiClient, snap.id) + if snap.id in self._vm_snapshot_ids: + self._vm_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + self._wait_until_gone( + lambda: any(item.id == snap.id for item in self._vm_snapshots_of(vm.id)), + "VM snapshot %s still listed after delete" % snap.id) + self._wait_ontap_snaps_gone(names) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_marker(token, msg="after VM snapshot delete") + + def _ontap_names_for_snap(self, names, snap, kind): + prefix = "cs" if kind == "volume" else "vm" + table = "snapshots" if kind == "volume" else "vm_snapshots" + db_id = tcu.get_db_id(self.dbConnection, table, snap.id) + suffix = "_%s%s" % (prefix, db_id) + return {name for name in names if str(name).endswith(suffix)} + + def _keep_vm_snapshot(self, snap, ontap_names, which): + """Leave this VM snapshot in place for step 17.""" + if snap.id in self._vm_snapshot_ids: + self._vm_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + kept = self.__class__.kept_ontap_snaps + if kept is None: + kept = set() + self.__class__.kept_ontap_snaps = kept + kept.update(names) + setattr(self.__class__, which, snap) + + def _adopt_kept_vm_snapshots(self): + """Hand the step 15 and 16 snapshots to this step's tearDown.""" + s1 = self.__class__.vm_snap_running + s2 = self.__class__.vm_snap_stopped + token_a = self.__class__.token_s1 + token_b = self.__class__.token_s2 + if s1 is None or s2 is None or not token_a or not token_b: + self.skipTest("VM snapshots from test_15 and test_16 are required") + ontap_names = set(self.__class__.kept_ontap_snaps or []) + self._vm_snapshot_ids.extend([s1.id, s2.id]) + self._created_ontap_snaps |= ontap_names + self.__class__.vm_snap_running = None + self.__class__.vm_snap_stopped = None + self.__class__.token_s1 = None + self.__class__.token_s2 = None + self.__class__.kept_ontap_snaps = set() + return s1, s2, token_a, token_b, ontap_names + + def _require_snapshot_vm(self, ssh=True): + if self.__class__.vm is None: + self.skipTest("VM not deployed — test_03 was skipped (no ready template)") + if ssh and self.__class__.vm_ssh is None: + self.skipTest("Guest SSH not set up — test_09 must pass first") + self._start_vm(self.__class__.vm.id) + return self.__class__.vm + + def _require_second_vm(self): + self._require_snapshot_vm() + if self.__class__.vm2 is None or self.__class__.vm2_ssh is None: + self.skipTest("Second VM not deployed — test_19 must pass first") + self._start_vm(self.__class__.vm2.id) + return self.__class__.vm2 + + def _vm_state(self, vm_id): + vms = self.apiClient.listVirtualMachines(_list_vms_cmd(vm_id)) + return vms[0].state if vms else None + + def _stop_vm(self, vm_id): + if (self._vm_state(vm_id) or "").lower() == "stopped": + return + cmd = stopVirtualMachineAPI.stopVirtualMachineCmd() + cmd.id = vm_id + cmd.forced = True + self.apiClient.stopVirtualMachine(cmd) + self._poll_vm_state(vm_id, "Stopped", timeout=600) + + def _start_vm(self, vm_id): + if (self._vm_state(vm_id) or "").lower() == "running": + return + cmd = startVirtualMachineAPI.startVirtualMachineCmd() + cmd.id = vm_id + self.apiClient.startVirtualMachine(cmd) + self._poll_vm_state(vm_id, "Running", timeout=900) + + def _root_volume(self, vm_id): + cmd = listVolumesAPI.listVolumesCmd() + cmd.virtualmachineid = vm_id + cmd.type = "ROOT" + cmd.listall = True + vols = self.apiClient.listVolumes(cmd) or [] + self.assertTrue(vols, "No ROOT volume found for VM %s" % vm_id) + return vols[0] + + def _guest(self, which="vm"): + host, port = getattr(self.__class__, "%s_ssh" % which) + return GuestSsh(host, port, self.guest_user, self.guest_password) + + def _write_marker(self, token, which="vm", disks=("ROOT",)): + with self._guest(which) as guest: + if "ROOT" in disks: + guest.write_root_marker(token) + if "DATA" in disks: + guest.write_data_marker(token) + + def _assert_marker(self, token, which="vm", disk="ROOT", msg=""): + with self._guest(which) as guest: + found = guest.read_data_marker() if disk == "DATA" else guest.read_root_marker() + self.assertEqual(found, token, "%s: %s marker on %s is %r, expected %r" + % (msg, disk, which, found, token)) + + def _ontap_snap_names(self): + return {s.get("name") for s in self.ontap.list_snapshots(self.__class__.pool.name)} + + def _assert_new_ontap_snaps(self, before, snaps, kind, msg): + """Return the FlexVol snapshot created for each CloudStack snapshot. + + A volume snapshot is stored as ``..._cs`` and a VM snapshot as + ``..._vm``. ONTAP snapshots from its own schedule, such as + ``hourly.*`` or ``Anti_ransomware_*``, are ignored. + """ + new = self._ontap_snap_names() - before + prefix = "cs" if kind == "volume" else "vm" + table = "snapshots" if kind == "volume" else "vm_snapshots" + matched = set() + missing = [] + for snap in snaps: + db_id = tcu.get_db_id(self.dbConnection, table, snap.id) + suffix = "_%s%s" % (prefix, db_id) + hits = sorted(name for name in new if name.endswith(suffix)) + if len(hits) != 1: + missing.append("%s (suffix %s, matches %s)" % (snap.id, suffix, hits)) + else: + matched.add(hits[0]) + self.assertFalse( + missing, + "%s: FlexVol snapshot missing for %s; new FlexVol snapshots: %s" + % (msg, missing, sorted(new))) + ignored = new - matched + if ignored: + logger.info("%s: ignoring FlexVol snapshots not created by this step: %s", + msg, sorted(ignored)) + self._created_ontap_snaps |= matched + return matched + + def _assert_ontap_snaps_present(self, names, msg): + missing = set(names) - self._ontap_snap_names() + self.assertFalse(missing, "%s: FlexVol snapshots missing: %s" % (msg, sorted(missing))) + + def _wait_ontap_snaps_gone(self, names, timeout=180, fail=True): + names = set(names) + deadline = time.time() + timeout + left = names + while time.time() < deadline: + left = names & self._ontap_snap_names() + if not left: + return True + time.sleep(5) + if fail: + self.fail("FlexVol snapshots %s still present after %ds" % (sorted(left), timeout)) + return False + + def _take_volume_snapshot(self, volume_id): + snap = Snapshot.create(self.apiClient, volume_id, + account=self.account.name, domainid=self.domain.id) + self._volume_snapshot_ids.append(snap.id) + self.assertEqual(snap.state, "BackedUp", + "Volume snapshot %s is %s, expected BackedUp" % (snap.id, snap.state)) + return snap + + def _take_vm_snapshot(self, vm_id): + snap = VmSnapshot.create(self.apiClient, vm_id, snapshotmemory="false", + name="ontap_%d" % random.randint(0, 999999)) + self._vm_snapshot_ids.append(snap.id) + self.assertEqual(snap.state, "Ready", + "VM snapshot %s is %s, expected Ready" % (snap.id, snap.state)) + return snap + + def _vm_snapshots_of(self, vm_id): + return VmSnapshot.list(self.apiClient, virtualmachineid=vm_id, listall=True) or [] + + def _volume_snapshot_listed(self, snap_id): + listed = Snapshot.list(self.apiClient, id=snap_id, listall=True) or [] + return any(getattr(item, "id", None) == snap_id for item in listed) + + def _wait_until_gone(self, still_there, msg, timeout=180): + deadline = time.time() + timeout + while time.time() < deadline and still_there(): + time.sleep(5) + self.assertFalse(still_there(), msg) + + def _volume_revert_flow(self, disk): + """Write A, snapshot, write B, stop, revert, start; expect A.""" + vm = self._require_snapshot_vm() + volume_id = (self.__class__.volume.id if disk == "DATA" + else self._root_volume(vm.id).id) + token_a = new_token("before-volume-snapshot") + token_b = new_token("after-volume-snapshot") + self._write_marker(token_a, disks=(disk,)) + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(volume_id) + new = self._assert_new_ontap_snaps(before, [snap], "volume", "%s volume snapshot" % disk) + self._write_marker(token_b, disks=(disk,)) + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, snap.id) + self._assert_ontap_snaps_present(new, "after revert") + self._start_vm(vm.id) + self._assert_marker(token_a, disk=disk, + msg="after revert (write after snapshot was %s)" % token_b) + # ================================================================== # Test steps # ================================================================== @@ -549,6 +1009,7 @@ def test_02_create_ontap_data_volume(self): self._data_lun_size(), detached_target, "ONTAP LUN size does not match detached grow target") + # ------------------------------------------------------------------ # Step 03 — Deploy a VM # ------------------------------------------------------------------ @@ -725,6 +1186,7 @@ def test_04_attach_volume_to_vm(self): self._data_lun_size(), initial_size, "Initial ONTAP LUN size should match the CloudStack volume size") + # ------------------------------------------------------------------ # Step 05 — Stop VM — LUN-maps should be removed # ------------------------------------------------------------------ @@ -768,6 +1230,8 @@ def test_05_stop_vm_lun_unmapped(self): "Data volume LUN should still exist in ONTAP FlexVol after VM stop" ) + # ------------------------------------------------------------------ + # ------------------------------------------------------------------ # Step 06 — Grow attached volume while VM is stopped # ------------------------------------------------------------------ @@ -856,7 +1320,6 @@ def test_07_shrink_rejected(self): self._data_lun_size(), self.__class__.target_volume_size, "ONTAP LUN size changed after rejected shrink") - # ------------------------------------------------------------------ # Step 08 — Start VM — LUN-maps should be re-created # ------------------------------------------------------------------ @@ -892,11 +1355,421 @@ def test_08_start_vm_lun_remapped(self): ) # ------------------------------------------------------------------ - # Step 09 — Detach volume from running VM + # Step 09 — Guest SSH access for snapshot checks + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_09_open_ssh_and_write_root_data_files(self): + """ + Prepare the running VM for snapshot checks. + Verifies: + - ROOT and DATA LUNs are both on the ONTAP test pool + - SSH login (ontap.cfg templateUsername/templatePassword) works + through a port forward on the network's source-NAT IP + - The DATA disk can be formatted, mounted and written + """ + vm = self._require_snapshot_vm(ssh=False) + if not self.__class__.network_id: + self.skipTest("Snapshot steps need an Advanced zone guest network for SSH") + pool_id = self.__class__.pool.id + root = self._root_volume(vm.id) + self.assertEqual(getattr(root, "storageid", None), pool_id, + "ROOT volume is on '%s', not the ONTAP test pool" + % getattr(root, "storage", None)) + vol_cmd = listVolumesAPI.listVolumesCmd() + vol_cmd.id = self.__class__.volume.id + vols = self.apiClient.listVolumes(vol_cmd) or [] + self.assertTrue(vols and getattr(vols[0], "virtualmachineid", None) == vm.id, + "DATA volume must be attached before the snapshot steps") + + host, port, _ = open_ssh_port(self.apiClient, vm.id, self.__class__.network_id) + self.__class__.vm_ssh = (host, port) + token = new_token("ssh-access-test") + self._write_marker(token, disks=("ROOT", "DATA")) + self._assert_marker(token, disk="ROOT", msg="guest access") + self._assert_marker(token, disk="DATA", msg="guest access") + + # ------------------------------------------------------------------ + # Steps 08-12 — Volume snapshots + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_10_root_volume_snapshot_create(self): + """ + VS-01: snapshot the ROOT volume of the running VM. + Verifies: state BackedUp; VM stays Running; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(self._root_volume(vm.id).id) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_new_ontap_snaps(before, [snap], "volume", "ROOT volume snapshot") + + @attr(tags=[TAG], required_hardware=True) + def test_11_data_volume_snapshot_create(self): + """ + VS-03: snapshot the attached DATA volume. + The snapshot is kept for step 14. + Verifies: state BackedUp; VM stays Running; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + token = new_token("data-s1") + self._write_marker(token, disks=("DATA",)) + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(self.__class__.volume.id) + self.assertEqual(self._vm_state(vm.id), "Running") + new = self._assert_new_ontap_snaps(before, [snap], "volume", "DATA volume snapshot") + self._keep_data_snapshot(snap, new, token) + + @attr(tags=[TAG], required_hardware=True) + def test_12_volume_revert_running_vm_rejected(self): + """ + VS-20: reverting a volume snapshot while the VM is Running fails. + Verifies: error says the Instance is not shut down; VM stays Running; + the FlexVol snapshot remains. + """ + vm = self._require_snapshot_vm() + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(self._root_volume(vm.id).id) + new = self._assert_new_ontap_snaps(before, [snap], "volume", "ROOT volume snapshot") + with self.assertRaises(Exception) as ctx: + Volume.revertToSnapshot(self.apiClient, snap.id) + self.assertIn("shutdown", str(ctx.exception).lower(), + "Unexpected error: %s" % ctx.exception) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_ontap_snaps_present(new, "after rejected revert") + + @attr(tags=[TAG], required_hardware=True) + def test_13_root_volume_revert(self): + """ + VS-12: write A, snapshot ROOT, write B, stop, revert, start. + Verifies: ROOT marker is A; the FlexVol snapshot survives the revert. + """ + self._volume_revert_flow("ROOT") + + @attr(tags=[TAG], required_hardware=True) + def test_14_data_volume_revert(self): + """ + Take a second DATA snapshot, revert to the step 11 snapshot, then to + this later one, and delete both while the VM is still running. + Verifies: + - After revert to the older snapshot, the DATA marker is the step 11 + token and the later snapshot is still listed + - After revert to the later snapshot, the DATA marker is that token + - Both snapshots and their FlexVol snapshots are gone; the VM stays + Running and the later marker is unchanged + """ + vm = self._require_snapshot_vm() + older, token_older, ontap_older = self._adopt_kept_data_snapshot() + token_latest = new_token("data-s2") + self._write_marker(token_latest, disks=("DATA",)) + before = self._ontap_snap_names() + latest = self._take_volume_snapshot(self.__class__.volume.id) + ontap_latest = self._assert_new_ontap_snaps( + before, [latest], "volume", "latest DATA snapshot") + self._write_marker(new_token("data-after-both"), disks=("DATA",)) + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, older.id) + self._start_vm(vm.id) + self._assert_marker(token_older, disk="DATA", + msg="after revert to the older snapshot") + self.assertTrue( + self._volume_snapshot_listed(latest.id), + "The later volume snapshot should still exist") + self._assert_ontap_snaps_present( + ontap_older | ontap_latest, "after revert to the older snapshot") + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, latest.id) + self._start_vm(vm.id) + self._assert_marker(token_latest, disk="DATA", + msg="after revert to the latest snapshot") + + self._delete_volume_snapshot_while_running( + vm, latest, ontap_latest, token_latest) + self._delete_volume_snapshot_while_running( + vm, older, ontap_older, token_latest) + + # ------------------------------------------------------------------ + # Steps 13-16 — VM snapshots + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_15_vm_snapshot_create_running(self): + """ + VM snapshot of the running VM (ROOT + DATA LUNs on one FlexVol). + Writes before-s1, then snapshots. The snapshot is kept for step 17. + Verifies: state Ready; VM stays Running; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + self.__class__.token_s1 = new_token("before-s1") + self._write_marker(self.__class__.token_s1, disks=("ROOT", "DATA")) + before = self._ontap_snap_names() + snap = self._take_vm_snapshot(vm.id) + self.assertEqual(self._vm_state(vm.id), "Running") + new = self._assert_new_ontap_snaps(before, [snap], "vm", "VM snapshot") + self._keep_vm_snapshot(snap, new, "vm_snap_running") + + @attr(tags=[TAG], required_hardware=True) + def test_16_vm_snapshot_create_stopped(self): + """ + VM snapshot of the stopped VM. + Writes before-s2 while Running, then stops and snapshots. + The snapshot is kept for step 17. + Verifies: state Ready; VM stays Stopped; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + self.__class__.token_s2 = new_token("before-s2") + self._write_marker(self.__class__.token_s2, disks=("ROOT", "DATA")) + self._stop_vm(vm.id) + before = self._ontap_snap_names() + snap = self._take_vm_snapshot(vm.id) + self.assertEqual(self._vm_state(vm.id), "Stopped") + new = self._assert_new_ontap_snaps(before, [snap], "vm", "VM snapshot (Stopped)") + self._keep_vm_snapshot(snap, new, "vm_snap_stopped") + + @attr(tags=[TAG], required_hardware=True) + def test_17_vm_snapshot_s1_running_s2_stopped_revert(self): + """ + VMS-09 (iSCSI ROOT + iSCSI DATA on the same pool): revert to the + running snapshot from step 15, then to the stopped snapshot from + step 16. + Verifies: + - After revert to S1: ROOT and DATA hold before-s1 + - After revert to S2: ROOT and DATA hold before-s2 + - Both FlexVol snapshots survive the reverts + - Both VM snapshots are then deleted while the VM is still running + """ + if self.__class__.vm is None: + self.skipTest("VM not deployed — test_03 was skipped (no ready template)") + if self.__class__.vm_ssh is None: + self.skipTest("Guest SSH not set up — test_09 must pass first") + s1, s2, token_a, token_b, ontap_names = self._adopt_kept_vm_snapshots() + vm = self.__class__.vm + + self._stop_vm(vm.id) + VmSnapshot.revertToSnapshot(self.apiClient, s1.id) + self._start_vm(vm.id) + self._assert_marker(token_a, disk="ROOT", msg="after revert to S1") + self._assert_marker(token_a, disk="DATA", msg="after revert to S1") + self.assertEqual(len(self._vm_snapshots_of(vm.id)), 2, "S2 should still be listed") + + self._stop_vm(vm.id) + VmSnapshot.revertToSnapshot(self.apiClient, s2.id) + self._start_vm(vm.id) + self._assert_marker(token_b, disk="ROOT", msg="after revert to S2") + self._assert_marker(token_b, disk="DATA", msg="after revert to S2") + self._assert_ontap_snaps_present(ontap_names, "after both reverts") + + self._delete_vm_snapshot_while_running( + vm, s1, self._ontap_names_for_snap(ontap_names, s1, "vm"), token_b) + self._delete_vm_snapshot_while_running( + vm, s2, self._ontap_names_for_snap(ontap_names, s2, "vm"), token_b) + + @attr(tags=[TAG], required_hardware=True) + def test_18_vm_snapshot_duplicate_requests(self): + """ + VMS-53: two VM snapshot requests for the same VM at the same time. + A rejected request is retried on its own; one that already succeeded + is not repeated. + Verifies: both succeed and are Ready; two new FlexVol snapshots. + """ + vm = self._require_snapshot_vm() + before = self._ontap_snap_names() + pending = [ + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_dup_a_%d" % random.randint(0, 99999)), + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_dup_b_%d" % random.randint(0, 99999)), + ] + ok = [] + last_errors = [] + attempts = 3 + for attempt in range(1, attempts + 1): + results = _run_concurrent(*pending) + last_errors = [] + still_pending = [] + retryable = True + for fn, (result, exc) in zip(pending, results): + if exc is None and result is not None: + ok.append(result) + self._vm_snapshot_ids.append(result.id) + continue + last_errors.append(exc) + if exc is None or not _is_rejected(exc): + retryable = False + still_pending.append(fn) + pending = still_pending + if not pending: + break + if not retryable or attempt == attempts: + break + logger.warning("duplicate VM snapshot rejected, retry %d/%d: %s", + attempt, attempts, last_errors) + time.sleep(15) + self.assertFalse(pending, + "Duplicate VM snapshot requests failed: %s" % last_errors) + self.assertEqual(len(ok), 2, "Expected both VM snapshots to succeed; " + "got %d" % len(ok)) + for snap in ok: + self.assertEqual(snap.state, "Ready") + self.assertEqual(snap.virtualmachineid, vm.id) + self._assert_new_ontap_snaps(before, ok, "vm", "duplicate VM snapshot requests") + + # ------------------------------------------------------------------ + # Steps 17-21 — Second VM on the same pool: isolation and concurrency + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_19_deploy_second_vm(self): + """ + Deploy a second VM whose ROOT LUN is on the same ONTAP pool. + Verifies: VM Running; ROOT on the test pool; SSH works. + """ + self._require_snapshot_vm() + cmd = deployVirtualMachineAPI.deployVirtualMachineCmd() + cmd.zoneid = self.zone.id + cmd.templateid = self.__class__.template_id + cmd.serviceofferingid = self.__class__.service_offering_id + cmd.account = self.account.name + cmd.domainid = self.domain.id + cmd.networkids = self.__class__.network_id + vm2 = self.apiClient.deployVirtualMachine(cmd) + self.__class__.vm2 = vm2 + self._poll_vm_state(vm2.id, "Running", timeout=900) + root = self._root_volume(vm2.id) + self.assertEqual(getattr(root, "storageid", None), self.__class__.pool.id, + "Second VM ROOT is not on the ONTAP test pool") + host, port, _ = open_ssh_port(self.apiClient, vm2.id, self.__class__.network_id) + self.__class__.vm2_ssh = (host, port) + token = new_token("second-vm-ssh-access-test") + self._write_marker(token, which="vm2") + self._assert_marker(token, which="vm2", msg="second VM access") + + @attr(tags=[TAG], required_hardware=True) + def test_20_volume_revert_leaves_other_vm_untouched(self): + """ + VS-28: two VMs on one FlexVol; revert VM-A's ROOT to its snapshot. + Verifies: VM-A is back at its snapshot; VM-B keeps data written after it. + """ + vm = self._require_snapshot_vm() + self._require_second_vm() + a1 = new_token("vm-a-before-snapshot") + a2 = new_token("vm-a-after-snapshot") + b1 = new_token("vm-b-before-snapshot") + b2 = new_token("vm-b-after-snapshot") + self._write_marker(a1) + self._write_marker(b1, which="vm2") + snap = self._take_volume_snapshot(self._root_volume(vm.id).id) + self._write_marker(a2) + self._write_marker(b2, which="vm2") + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, snap.id) + self._start_vm(vm.id) + self._assert_marker(a1, msg="VM-A after revert") + self._assert_marker(b2, which="vm2", msg="VM-B after VM-A's revert") + + @attr(tags=[TAG], required_hardware=True) + def test_21_concurrent_vm_snapshots_two_vms(self): + """ + VMS-55: VM snapshots on two VMs of the same iSCSI pool at the same time. + Verifies: both Ready and owned by their VMs; two distinct new FlexVol + snapshots. + """ + vm = self._require_snapshot_vm() + vm2 = self._require_second_vm() + before = self._ontap_snap_names() + results = _run_concurrent( + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_cc_a_%d" % random.randint(0, 99999)), + lambda: VmSnapshot.create(self.apiClient, vm2.id, snapshotmemory="false", + name="ontap_cc_b_%d" % random.randint(0, 99999)), + ) + self._vm_snapshot_ids.extend(s.id for s, _ in results if s is not None) + errors = [e for _, e in results if e is not None] + self.assertFalse(errors, "Concurrent VM snapshots failed: %s" % errors) + snap_a, snap_b = results[0][0], results[1][0] + self.assertEqual(snap_a.state, "Ready") + self.assertEqual(snap_b.state, "Ready") + self.assertEqual(snap_a.virtualmachineid, vm.id) + self.assertEqual(snap_b.virtualmachineid, vm2.id) + self._assert_new_ontap_snaps(before, [snap_a, snap_b], "vm", "concurrent VM snapshots") + + @attr(tags=[TAG], required_hardware=True) + def test_22_concurrent_volume_snapshot_and_revert(self): + """ + Volume snapshot of VM-A's ROOT while VM-B's ROOT reverts, at the same time. + A rejected snapshot or revert is retried on its own. + Verifies: VM-A snapshot BackedUp with its own FlexVol snapshot; VM-B is + back at its snapshot; VM-A data unchanged. + """ + vm = self._require_snapshot_vm() + vm2 = self._require_second_vm() + a1 = new_token("vm-a-current-data") + b1 = new_token("vm-b-before-snapshot") + b2 = new_token("vm-b-after-snapshot") + self._write_marker(a1) + self._write_marker(b1, which="vm2") + snap_b = self._take_volume_snapshot(self._root_volume(vm2.id).id) + self._write_marker(b2, which="vm2") + self._stop_vm(vm2.id) + before = self._ontap_snap_names() + root_a = self._root_volume(vm.id).id + + snap_a = _snapshot_and_revert_with_retry( + lambda: Snapshot.create(self.apiClient, root_a, + account=self.account.name, domainid=self.domain.id), + lambda: Volume.revertToSnapshot(self.apiClient, snap_b.id), + ) + self._volume_snapshot_ids.append(snap_a.id) + self.assertEqual(snap_a.state, "BackedUp") + self._assert_new_ontap_snaps(before, [snap_a], "volume", "VM-A snapshot during VM-B revert") + + self._start_vm(vm2.id) + self._assert_marker(b1, which="vm2", msg="VM-B after revert") + self._assert_marker(a1, msg="VM-A during VM-B revert") + + @attr(tags=[TAG], required_hardware=True) + def test_23_concurrent_vm_snapshot_and_revert(self): + """ + VM snapshot of VM-A while VM-B reverts to its VM snapshot, at the same time. + A rejected snapshot or revert is retried on its own. + Verifies: VM-A snapshot Ready with its own FlexVol snapshot; VM-B is + back at its snapshot; VM-A data unchanged. + """ + vm = self._require_snapshot_vm() + vm2 = self._require_second_vm() + a1 = new_token("vm-a-current-data") + b1 = new_token("vm-b-before-snapshot") + b2 = new_token("vm-b-after-snapshot") + self._write_marker(a1) + self._write_marker(b1, which="vm2") + snap_b = self._take_vm_snapshot(vm2.id) + self._write_marker(b2, which="vm2") + self._stop_vm(vm2.id) + before = self._ontap_snap_names() + + snap_a = _snapshot_and_revert_with_retry( + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_ccr_%d" % random.randint(0, 99999)), + lambda: VmSnapshot.revertToSnapshot(self.apiClient, snap_b.id), + ) + self._vm_snapshot_ids.append(snap_a.id) + self.assertEqual(snap_a.state, "Ready") + self._assert_new_ontap_snaps(before, [snap_a], "vm", "VM-A snapshot during VM-B revert") + + self._start_vm(vm2.id) + self._assert_marker(b1, which="vm2", msg="VM-B after revert") + self._assert_marker(a1, msg="VM-A during VM-B revert") + + # ------------------------------------------------------------------ + # Step 24 — Detach volume from running VM # ------------------------------------------------------------------ @attr(tags=["iscsi_vm_workflow"], required_hardware=True) - def test_09_detach_volume_from_vm(self): + def test_24_detach_volume_from_vm(self): """ Detach the iSCSI data volume from the running VM. Verifies: @@ -955,11 +1828,13 @@ def test_09_detach_volume_from_vm(self): ) # ------------------------------------------------------------------ - # Step 10 — Grow detached data volume + + # ------------------------------------------------------------------ + # Step 25 — Grow detached data volume # ------------------------------------------------------------------ @attr(tags=["iscsi_vm_workflow"], required_hardware=True) - def test_10_grow_detached_volume(self): + def test_25_grow_detached_volume(self): """Grow the materialized LUN after it has been detached.""" self.assertIsNotNone(self.__class__.volume, "Volume absent — test_02 must pass first") @@ -978,11 +1853,43 @@ def test_10_grow_detached_volume(self): self.__class__.target_volume_size = detached_target # ------------------------------------------------------------------ - # Step 11 — Destroy VM and clean up pool + # Step 26 — Destroy the second VM while it still has a VM snapshot + # ------------------------------------------------------------------ + + @attr(tags=["iscsi_vm_workflow"], required_hardware=True) + def test_26_destroy_vm_with_vm_snapshot(self): + """ + Destroy (expunge) the second VM while it still has a VM snapshot. + Verifies: destroy succeeds; the VM snapshot is no longer listed; its + FlexVol snapshot is removed from ONTAP. + """ + self._require_snapshot_vm() + vm2 = self._require_second_vm() + before = self._ontap_snap_names() + snap = self._take_vm_snapshot(vm2.id) + new = self._assert_new_ontap_snaps(before, [snap], "vm", "second VM snapshot") + + cmd = destroyVirtualMachineAPI.destroyVirtualMachineCmd() + cmd.id = vm2.id + cmd.expunge = True + self.apiClient.destroyVirtualMachine(cmd) + self.__class__.vm2 = None + self.__class__.vm2_ssh = None + self._vm_snapshot_ids.remove(snap.id) + + deadline = time.time() + 300 + while time.time() < deadline and self._vm_snapshots_of(vm2.id): + time.sleep(10) + self.assertFalse(self._vm_snapshots_of(vm2.id), + "VM snapshot still listed after destroying its VM") + self._wait_ontap_snaps_gone(new) + + # ------------------------------------------------------------------ + # Step 27 — Destroy VM and clean up pool # ------------------------------------------------------------------ @attr(tags=["iscsi_vm_workflow"], required_hardware=True) - def test_11_destroy_vm_and_cleanup(self): + def test_27_destroy_vm_and_cleanup(self): """ Destroy the VM (with expunge), delete the data volume, force-delete the ONTAP pool, and delete the guest network created for this suite. @@ -997,6 +1904,7 @@ def test_11_destroy_vm_and_cleanup(self): - ONTAP: all per-host igroups deleted - CloudStack: guest network deleted (Advanced zones only) """ + self._destroy_second_vm() pool = self.__class__.pool vol = self.__class__.volume diff --git a/test/integration/plugins/ontap/nfs3/instance/test_vm_volume_attach.py b/test/integration/plugins/ontap/nfs3/instance/test_vm_volume_attach.py index af63b4cd9d3f..d3450949baf2 100644 --- a/test/integration/plugins/ontap/nfs3/instance/test_vm_volume_attach.py +++ b/test/integration/plugins/ontap/nfs3/instance/test_vm_volume_attach.py @@ -19,14 +19,14 @@ Sequential workflow integration tests for NetApp ONTAP data volume lifecycle with a running virtual machine. -Tests are numbered test_01 ... test_11, plus test_03a and test_03b, and must +Tests are numbered test_01 ... test_27, plus test_03a and test_03b, and must run in that order. Each step builds on the shared state established by the previous step. The VM is deployed with a compute offering whose storage tags match the pool, so its ROOT volume lands on the ONTAP pool and exercises the primary template cache (seed / reuse / survive VM delete) alongside the data-volume -resize workflow. +resize and snapshot workflow. Workflow: 01 Create NFS3 primary storage pool on ONTAP (tagged -tmpl-cache) @@ -40,11 +40,39 @@ 06 Grow the attached volume and its qcow2 file while the VM is stopped 07 Verify shrinking the volume is rejected without changing its size 08 Start the VM — VM Running; volume still attached; FlexVol online - 09 Detach the ONTAP data volume from the VM - 10 Grow the detached data volume - 11 Destroy VM (cache still survives); delete ONTAP volume; enter + 09 Open an SSH port forward to the VM; write ROOT and DATA files + 10 Snapshot the ROOT volume of the running VM + 11 Snapshot the DATA volume of the running VM (kept for step 14) + 12 Revert a volume snapshot while the VM is Running — rejected + 13 ROOT volume revert: write before-snapshot, snapshot, write + after-snapshot, revert, file is back to before-snapshot + 14 DATA: snapshot again, revert to the step-11 snapshot then to this + one, then delete both while the VM is still running + 15 Write before-s1; VM snapshot of the running VM (kept for step 17) + 16 Write before-s2, stop, VM snapshot (kept for step 17) + 17 Revert to the step-15 snapshot, then to the step-16 snapshot, then + delete both while the VM is still running + 18 Two VM snapshot requests for the same VM at once — both succeed + 19 Deploy a second VM whose ROOT disk is on the same pool + 20 Revert VM-A's ROOT; VM-B's file is unchanged + 21 VM snapshots of both VMs at the same time + 22 Volume snapshot of VM-A while VM-B's ROOT reverts + 23 VM snapshot of VM-A while VM-B reverts to its VM snapshot + 24 Detach the ONTAP data volume from the VM + 25 Grow the detached data volume + 26 Expunge the second VM while it still has a VM snapshot + 27 Destroy VM (cache still survives); delete ONTAP volume; enter maintenance; force-delete pool +Snapshot steps only create and verify; tearDown deletes every snapshot the +step created and waits for its FlexVol snapshot to go. Step 11 keeps its +DATA snapshot so step 14 can revert to it and then delete both DATA +snapshots while the VM is running. Steps 15 and 16 keep their VM snapshots +so step 17 can revert to them and then delete both while the VM is running. + +Guest login for the CentOS 5.5 template comes from ontap.cfg +(cloudstack.templateUsername / templatePassword); see guest_ssh.py. + Prerequisites: - CloudStack management server with the NetApp ONTAP plugin deployed - KVM cluster registered in CloudStack with at least one executable template @@ -62,6 +90,7 @@ import base64 import logging import random +import threading import time import unittest @@ -85,9 +114,10 @@ stopVirtualMachine as stopVirtualMachineAPI, updateStoragePool as updateStoragePoolAPI, ) -from marvin.lib.base import ServiceOffering, StoragePool +from marvin.lib.base import ServiceOffering, Snapshot, StoragePool, VmSnapshot, Volume from marvin.lib.common import list_storage_pools +from guest_ssh import GuestSsh, guest_credentials, new_token, open_ssh_port from ontap_test_base import OntapRestClient, OntapTestBase, _parse_pool_details, get_datacenter_config from helpers import template_cache_util as tcu from helpers.template_cache_base import ( @@ -99,6 +129,8 @@ logger = logging.getLogger("TestOntapVMVolumeAttach") +TAG = "vm_volume_workflow" + # --------------------------------------------------------------------------- # Test data @@ -119,12 +151,14 @@ class TestData: DETAIL_STORAGE_IP = "storageIP" ONTAP_MIN_VOLUME_SIZE = 1677721600 + # Used when ontap.cfg has neither templateCacheCapacitybytes nor capacitybytes. + DEFAULT_POOL_SIZE = 50 * 1024 ** 3 def __init__(self, storage_ip, svm_name, username, password, protocol="NFS3", scope="CLUSTER", provider="NetApp ONTAP", tags="ontap-nfs3", capacitybytes=None): - if capacitybytes is None: - capacitybytes = TestData.ONTAP_MIN_VOLUME_SIZE * 2 + if not capacitybytes: + capacitybytes = TestData.DEFAULT_POOL_SIZE encoded_password = base64.b64encode(password.encode()).decode() self.testdata = { TestData.ontap: { @@ -184,6 +218,18 @@ class TestOntapVMVolumeAttach(OntapTestBase): initial_volume_size = None target_volume_size = None volume_path = None + vm_ssh = None # (public_ip, public_port) forwarded to vm:22 + vm2_ssh = None + vm_snap_running = None # step 15 snapshot, reused by step 17 + vm_snap_stopped = None # step 16 snapshot, reused by step 17 + token_s1 = None + token_s2 = None + kept_ontap_snaps = None + data_snap_older = None # step 11 DATA snapshot, reused by step 14 + data_token_older = None + kept_data_ontap_snaps = None + guest_user = "root" + guest_password = "password" _vol_name_prefix = "OntapVMVol" @@ -230,6 +276,8 @@ def setUpClass(cls): cls._setup_cloudstack_resources(config, cls.testdata[TestData.account]) + cls.guest_user, cls.guest_password = guest_credentials(config) + # Discover a suitable user KVM template in the zone (must be fully # downloaded; system-type templates are excluded as they cannot be # deployed as user VMs). @@ -294,6 +342,8 @@ def setUpClass(cls): @classmethod def tearDownClass(cls): """Destroy the VMs first, then delegate pool/volume cleanup to super.""" + cls._delete_kept_data_snapshot() + cls._delete_kept_vm_snapshots() for vm in (cls.vm2, cls.vm): if vm is None: continue @@ -381,6 +431,339 @@ def _volume_state(self, vol_id): vols = self.apiClient.listVolumes(cmd) return vols[0].state if vols else "unknown" + # ---- snapshot step helpers (steps 09-23) --------------------------- + + def setUp(self): + super(TestOntapVMVolumeAttach, self).setUp() + self._volume_snapshot_ids = [] + self._vm_snapshot_ids = [] + self._extra_volume_ids = [] + self._created_ontap_snaps = set() + + def tearDown(self): + """ + Delete the snapshots and extra volumes a step created, then check + their FlexVol snapshots are gone from ONTAP. + """ + for snap_id in self._vm_snapshot_ids: + try: + VmSnapshot.deleteVMSnapshot(self.apiClient, snap_id) + except Exception as exc: + logger.warning("tearDown: VM snapshot %s: %s" % (snap_id, exc)) + for snap_id in self._volume_snapshot_ids: + try: + Snapshot({"id": snap_id}).delete(self.apiClient) + except Exception as exc: + logger.warning("tearDown: volume snapshot %s: %s" % (snap_id, exc)) + for vol_id in self._extra_volume_ids: + try: + cmd = deleteVolumeAPI.deleteVolumeCmd() + cmd.id = vol_id + self.apiClient.deleteVolume(cmd) + except Exception as exc: + logger.warning("tearDown: volume %s: %s" % (vol_id, exc)) + if self._created_ontap_snaps and not self._wait_ontap_snaps_gone( + self._created_ontap_snaps, fail=False): + logger.warning("tearDown: FlexVol snapshots still present: %s" + % sorted(self._created_ontap_snaps)) + super(TestOntapVMVolumeAttach, self).tearDown() + + @classmethod + def _destroy_second_vm(cls): + if cls.vm2 is None: + return + try: + cmd = destroyVirtualMachineAPI.destroyVirtualMachineCmd() + cmd.id = cls.vm2.id + cmd.expunge = True + cls.apiClient.destroyVirtualMachine(cmd) + except Exception as exc: + logger.warning("Could not destroy second VM %s: %s" % (cls.vm2.id, exc)) + cls.vm2 = None + cls.vm2_ssh = None + + @classmethod + def _delete_kept_vm_snapshots(cls): + """Delete the step 15/16 snapshots when step 17 did not take them over.""" + for snap in (cls.vm_snap_running, cls.vm_snap_stopped): + if snap is None: + continue + try: + VmSnapshot.deleteVMSnapshot(cls.apiClient, snap.id) + except Exception as exc: + logger.warning("tearDownClass: VM snapshot %s: %s" % (snap.id, exc)) + cls.vm_snap_running = None + cls.vm_snap_stopped = None + cls.token_s1 = None + cls.token_s2 = None + cls.kept_ontap_snaps = set() + + @classmethod + def _delete_kept_data_snapshot(cls): + """Delete the step 11 DATA snapshot when step 14 did not take it over.""" + snap = cls.data_snap_older + if snap is None: + return + try: + Snapshot({"id": snap.id}).delete(cls.apiClient) + except Exception as exc: + logger.warning("tearDownClass: volume snapshot %s: %s" % (snap.id, exc)) + cls.data_snap_older = None + cls.data_token_older = None + cls.kept_data_ontap_snaps = set() + + def _keep_data_snapshot(self, snap, ontap_names, token): + """Leave the step 11 DATA snapshot in place for step 14.""" + if snap.id in self._volume_snapshot_ids: + self._volume_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + self.__class__.data_snap_older = snap + self.__class__.data_token_older = token + self.__class__.kept_data_ontap_snaps = names + + def _adopt_kept_data_snapshot(self): + """Hand the step 11 DATA snapshot to step 14.""" + snap = self.__class__.data_snap_older + token = self.__class__.data_token_older + if snap is None or not token: + self.skipTest("DATA snapshot from test_11 is required") + ontap_names = set(self.__class__.kept_data_ontap_snaps or []) + self._volume_snapshot_ids.append(snap.id) + self._created_ontap_snaps |= ontap_names + self.__class__.data_snap_older = None + self.__class__.data_token_older = None + self.__class__.kept_data_ontap_snaps = set() + return snap, token, ontap_names + + def _delete_volume_snapshot_while_running(self, vm, snap, ontap_names, token): + """Delete a volume snapshot while its VM is still running.""" + Snapshot({"id": snap.id}).delete(self.apiClient) + if snap.id in self._volume_snapshot_ids: + self._volume_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + self._wait_until_gone( + lambda: self._volume_snapshot_listed(snap.id), + "Volume snapshot %s still listed after delete" % snap.id) + self._wait_ontap_snaps_gone(names) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_marker(token, disk="DATA", msg="after volume snapshot delete") + + def _delete_vm_snapshot_while_running(self, vm, snap, ontap_names, token): + """Delete a VM snapshot while that VM is still running.""" + VmSnapshot.deleteVMSnapshot(self.apiClient, snap.id) + if snap.id in self._vm_snapshot_ids: + self._vm_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + self._wait_until_gone( + lambda: any(item.id == snap.id for item in self._vm_snapshots_of(vm.id)), + "VM snapshot %s still listed after delete" % snap.id) + self._wait_ontap_snaps_gone(names) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_marker(token, msg="after VM snapshot delete") + + def _ontap_names_for_snap(self, names, snap, kind): + prefix = "cs" if kind == "volume" else "vm" + table = "snapshots" if kind == "volume" else "vm_snapshots" + db_id = tcu.get_db_id(self.dbConnection, table, snap.id) + suffix = "_%s%s" % (prefix, db_id) + return {name for name in names if str(name).endswith(suffix)} + + def _keep_vm_snapshot(self, snap, ontap_names, which): + """Leave this VM snapshot in place for step 17.""" + if snap.id in self._vm_snapshot_ids: + self._vm_snapshot_ids.remove(snap.id) + names = set(ontap_names) + self._created_ontap_snaps -= names + kept = self.__class__.kept_ontap_snaps + if kept is None: + kept = set() + self.__class__.kept_ontap_snaps = kept + kept.update(names) + setattr(self.__class__, which, snap) + + def _adopt_kept_vm_snapshots(self): + """Hand the step 15 and 16 snapshots to this step's tearDown.""" + s1 = self.__class__.vm_snap_running + s2 = self.__class__.vm_snap_stopped + token_a = self.__class__.token_s1 + token_b = self.__class__.token_s2 + if s1 is None or s2 is None or not token_a or not token_b: + self.skipTest("VM snapshots from test_15 and test_16 are required") + ontap_names = set(self.__class__.kept_ontap_snaps or []) + self._vm_snapshot_ids.extend([s1.id, s2.id]) + self._created_ontap_snaps |= ontap_names + self.__class__.vm_snap_running = None + self.__class__.vm_snap_stopped = None + self.__class__.token_s1 = None + self.__class__.token_s2 = None + self.__class__.kept_ontap_snaps = set() + return s1, s2, token_a, token_b, ontap_names + + def _require_snapshot_vm(self, ssh=True): + if self.__class__.vm is None: + self.skipTest("VM not deployed — test_03 was skipped (no ready template)") + if ssh and self.__class__.vm_ssh is None: + self.skipTest("Guest SSH not set up — test_09 must pass first") + self._start_vm(self.__class__.vm.id) + return self.__class__.vm + + def _require_second_vm(self): + self._require_snapshot_vm() + if self.__class__.vm2 is None or self.__class__.vm2_ssh is None: + self.skipTest("Second VM not deployed — test_19 must pass first") + self._start_vm(self.__class__.vm2.id) + return self.__class__.vm2 + + def _vm_state(self, vm_id): + vms = self.apiClient.listVirtualMachines(_list_vms_cmd(vm_id)) + return vms[0].state if vms else None + + def _stop_vm(self, vm_id): + if (self._vm_state(vm_id) or "").lower() == "stopped": + return + cmd = stopVirtualMachineAPI.stopVirtualMachineCmd() + cmd.id = vm_id + cmd.forced = True + self.apiClient.stopVirtualMachine(cmd) + self._poll_vm_state(vm_id, "Stopped", timeout=600) + + def _start_vm(self, vm_id): + if (self._vm_state(vm_id) or "").lower() == "running": + return + cmd = startVirtualMachineAPI.startVirtualMachineCmd() + cmd.id = vm_id + self.apiClient.startVirtualMachine(cmd) + self._poll_vm_state(vm_id, "Running", timeout=900) + + def _root_volume(self, vm_id): + cmd = listVolumesAPI.listVolumesCmd() + cmd.virtualmachineid = vm_id + cmd.type = "ROOT" + cmd.listall = True + vols = self.apiClient.listVolumes(cmd) or [] + self.assertTrue(vols, "No ROOT volume found for VM %s" % vm_id) + return vols[0] + + def _guest(self, which="vm"): + host, port = getattr(self.__class__, "%s_ssh" % which) + return GuestSsh(host, port, self.guest_user, self.guest_password) + + def _write_marker(self, token, which="vm", disks=("ROOT",)): + with self._guest(which) as guest: + if "ROOT" in disks: + guest.write_root_marker(token) + if "DATA" in disks: + guest.write_data_marker(token) + + def _assert_marker(self, token, which="vm", disk="ROOT", msg=""): + with self._guest(which) as guest: + found = guest.read_data_marker() if disk == "DATA" else guest.read_root_marker() + self.assertEqual(found, token, "%s: %s marker on %s is %r, expected %r" + % (msg, disk, which, found, token)) + + def _ontap_snap_names(self): + return {s.get("name") for s in self.ontap.list_snapshots(self.__class__.pool.name)} + + def _assert_new_ontap_snaps(self, before, snaps, kind, msg): + """Return the FlexVol snapshot created for each CloudStack snapshot. + + A volume snapshot is stored as ``..._cs`` and a VM snapshot as + ``..._vm``. ONTAP snapshots from its own schedule, such as + ``hourly.*`` or ``Anti_ransomware_*``, are ignored. + """ + new = self._ontap_snap_names() - before + prefix = "cs" if kind == "volume" else "vm" + table = "snapshots" if kind == "volume" else "vm_snapshots" + matched = set() + missing = [] + for snap in snaps: + db_id = tcu.get_db_id(self.dbConnection, table, snap.id) + suffix = "_%s%s" % (prefix, db_id) + hits = sorted(name for name in new if name.endswith(suffix)) + if len(hits) != 1: + missing.append("%s (suffix %s, matches %s)" % (snap.id, suffix, hits)) + else: + matched.add(hits[0]) + self.assertFalse( + missing, + "%s: FlexVol snapshot missing for %s; new FlexVol snapshots: %s" + % (msg, missing, sorted(new))) + ignored = new - matched + if ignored: + logger.info("%s: ignoring FlexVol snapshots not created by this step: %s", + msg, sorted(ignored)) + self._created_ontap_snaps |= matched + return matched + + def _assert_ontap_snaps_present(self, names, msg): + missing = set(names) - self._ontap_snap_names() + self.assertFalse(missing, "%s: FlexVol snapshots missing: %s" % (msg, sorted(missing))) + + def _wait_ontap_snaps_gone(self, names, timeout=180, fail=True): + names = set(names) + deadline = time.time() + timeout + left = names + while time.time() < deadline: + left = names & self._ontap_snap_names() + if not left: + return True + time.sleep(5) + if fail: + self.fail("FlexVol snapshots %s still present after %ds" % (sorted(left), timeout)) + return False + + def _take_volume_snapshot(self, volume_id): + snap = Snapshot.create(self.apiClient, volume_id, + account=self.account.name, domainid=self.domain.id) + self._volume_snapshot_ids.append(snap.id) + self.assertEqual(snap.state, "BackedUp", + "Volume snapshot %s is %s, expected BackedUp" % (snap.id, snap.state)) + return snap + + def _take_vm_snapshot(self, vm_id): + snap = VmSnapshot.create(self.apiClient, vm_id, snapshotmemory="false", + name="ontap_%d" % random.randint(0, 999999)) + self._vm_snapshot_ids.append(snap.id) + self.assertEqual(snap.state, "Ready", + "VM snapshot %s is %s, expected Ready" % (snap.id, snap.state)) + return snap + + def _vm_snapshots_of(self, vm_id): + return VmSnapshot.list(self.apiClient, virtualmachineid=vm_id, listall=True) or [] + + def _volume_snapshot_listed(self, snap_id): + listed = Snapshot.list(self.apiClient, id=snap_id, listall=True) or [] + return any(getattr(item, "id", None) == snap_id for item in listed) + + def _wait_until_gone(self, still_there, msg, timeout=180): + deadline = time.time() + timeout + while time.time() < deadline and still_there(): + time.sleep(5) + self.assertFalse(still_there(), msg) + + def _volume_revert_flow(self, disk): + """Write A, snapshot, write B, stop, revert, start; expect A.""" + vm = self._require_snapshot_vm() + volume_id = (self.__class__.volume.id if disk == "DATA" + else self._root_volume(vm.id).id) + token_a = new_token("before-volume-snapshot") + token_b = new_token("after-volume-snapshot") + self._write_marker(token_a, disks=(disk,)) + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(volume_id) + new = self._assert_new_ontap_snaps(before, [snap], "volume", "%s volume snapshot" % disk) + self._write_marker(token_b, disks=(disk,)) + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, snap.id) + self._assert_ontap_snaps_present(new, "after revert") + self._start_vm(vm.id) + self._assert_marker(token_a, disk=disk, + msg="after revert (write after snapshot was %s)" % token_b) + # ================================================================== # Test steps # ================================================================== @@ -650,6 +1033,7 @@ def test_04_attach_volume_to_vm(self): "Initial qcow2 virtual size should match the CloudStack volume size" ) + # ------------------------------------------------------------------ # Step 05 - Stop VM — export policy must be retained # ------------------------------------------------------------------ @@ -708,6 +1092,8 @@ def test_05_stop_vm_export_retained(self): "got virtualmachineid=%s" % (vm.id, vol_vmid) ) + # ------------------------------------------------------------------ + # ------------------------------------------------------------------ # Step 06 - Grow attached volume while VM is stopped # ------------------------------------------------------------------ @@ -806,7 +1192,6 @@ def test_07_shrink_rejected(self): backend_size, self.__class__.target_volume_size, "qcow2 virtual size changed after rejected shrink") - # ------------------------------------------------------------------ # Step 08 - Start VM — volume accessible; FlexVol online # ------------------------------------------------------------------ @@ -870,12 +1255,420 @@ def test_08_start_vm_volume_accessible(self): self.__class__.target_volume_size, "KVM data-disk capacity does not reflect the NFS3 resize") + + # ------------------------------------------------------------------ + # Step 09 - Guest SSH access for snapshot checks + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_09_open_ssh_and_write_root_data_files(self): + """ + Prepare the running VM for snapshot checks. + Verifies: + - ROOT and DATA volumes are both on the ONTAP test pool + - SSH login (ontap.cfg templateUsername/templatePassword) works + through a port forward on the network's source-NAT IP + - The DATA disk can be formatted, mounted and written + """ + vm = self._require_snapshot_vm(ssh=False) + if not self.__class__.network_id: + self.skipTest("Snapshot steps need an Advanced zone guest network for SSH") + pool_id = self.__class__.pool.id + root = self._root_volume(vm.id) + self.assertEqual(getattr(root, "storageid", None), pool_id, + "ROOT volume is on '%s', not the ONTAP test pool" + % getattr(root, "storage", None)) + vols = self.apiClient.listVolumes(_list_vols_cmd(self.__class__.volume.id)) or [] + self.assertTrue(vols and getattr(vols[0], "virtualmachineid", None) == vm.id, + "DATA volume must be attached before the snapshot steps") + + host, port, _ = open_ssh_port(self.apiClient, vm.id, self.__class__.network_id) + self.__class__.vm_ssh = (host, port) + token = new_token("ssh-access-test") + self._write_marker(token, disks=("ROOT", "DATA")) + self._assert_marker(token, disk="ROOT", msg="guest access") + self._assert_marker(token, disk="DATA", msg="guest access") + # ------------------------------------------------------------------ - # Step 09 - Detach ONTAP data volume from the VM + # Steps 08-12 - Volume snapshots + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_10_root_volume_snapshot_create(self): + """ + VS-01: snapshot the ROOT volume of the running VM. + Verifies: state BackedUp; VM stays Running; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(self._root_volume(vm.id).id) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_new_ontap_snaps(before, [snap], "volume", "ROOT volume snapshot") + + @attr(tags=[TAG], required_hardware=True) + def test_11_data_volume_snapshot_create(self): + """ + VS-03: snapshot the attached DATA volume. + The snapshot is kept for step 14. + Verifies: state BackedUp; VM stays Running; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + token = new_token("data-s1") + self._write_marker(token, disks=("DATA",)) + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(self.__class__.volume.id) + self.assertEqual(self._vm_state(vm.id), "Running") + new = self._assert_new_ontap_snaps(before, [snap], "volume", "DATA volume snapshot") + self._keep_data_snapshot(snap, new, token) + + @attr(tags=[TAG], required_hardware=True) + def test_12_volume_revert_running_vm_rejected(self): + """ + VS-19: reverting a volume snapshot while the VM is Running fails. + Verifies: error says the Instance is not shut down; VM stays Running; + the FlexVol snapshot remains. + """ + vm = self._require_snapshot_vm() + before = self._ontap_snap_names() + snap = self._take_volume_snapshot(self._root_volume(vm.id).id) + new = self._assert_new_ontap_snaps(before, [snap], "volume", "ROOT volume snapshot") + with self.assertRaises(Exception) as ctx: + Volume.revertToSnapshot(self.apiClient, snap.id) + self.assertIn("shutdown", str(ctx.exception).lower(), + "Unexpected error: %s" % ctx.exception) + self.assertEqual(self._vm_state(vm.id), "Running") + self._assert_ontap_snaps_present(new, "after rejected revert") + + @attr(tags=[TAG], required_hardware=True) + def test_13_root_volume_revert(self): + """ + VS-11: write A, snapshot ROOT, write B, stop, revert, start. + Verifies: ROOT marker is A; the FlexVol snapshot survives the revert. + """ + self._volume_revert_flow("ROOT") + + @attr(tags=[TAG], required_hardware=True) + def test_14_data_volume_revert(self): + """ + Take a second DATA snapshot, revert to the step 11 snapshot, then to + this later one, and delete both while the VM is still running. + Verifies: + - After revert to the older snapshot, the DATA marker is the step 11 + token and the later snapshot is still listed + - After revert to the later snapshot, the DATA marker is that token + - Both snapshots and their FlexVol snapshots are gone; the VM stays + Running and the later marker is unchanged + """ + vm = self._require_snapshot_vm() + older, token_older, ontap_older = self._adopt_kept_data_snapshot() + token_latest = new_token("data-s2") + self._write_marker(token_latest, disks=("DATA",)) + before = self._ontap_snap_names() + latest = self._take_volume_snapshot(self.__class__.volume.id) + ontap_latest = self._assert_new_ontap_snaps( + before, [latest], "volume", "latest DATA snapshot") + self._write_marker(new_token("data-after-both"), disks=("DATA",)) + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, older.id) + self._start_vm(vm.id) + self._assert_marker(token_older, disk="DATA", + msg="after revert to the older snapshot") + self.assertTrue( + self._volume_snapshot_listed(latest.id), + "The later volume snapshot should still exist") + self._assert_ontap_snaps_present( + ontap_older | ontap_latest, "after revert to the older snapshot") + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, latest.id) + self._start_vm(vm.id) + self._assert_marker(token_latest, disk="DATA", + msg="after revert to the latest snapshot") + + self._delete_volume_snapshot_while_running( + vm, latest, ontap_latest, token_latest) + self._delete_volume_snapshot_while_running( + vm, older, ontap_older, token_latest) + + # ------------------------------------------------------------------ + # Steps 13-16 - VM snapshots + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_15_vm_snapshot_create_running(self): + """ + VMS-01: VM snapshot of the running VM (ROOT + DATA on one FlexVol). + Writes before-s1, then snapshots. The snapshot is kept for step 17. + Verifies: state Ready; VM stays Running; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + self.__class__.token_s1 = new_token("before-s1") + self._write_marker(self.__class__.token_s1, disks=("ROOT", "DATA")) + before = self._ontap_snap_names() + snap = self._take_vm_snapshot(vm.id) + self.assertEqual(self._vm_state(vm.id), "Running") + new = self._assert_new_ontap_snaps(before, [snap], "vm", "VM snapshot") + self._keep_vm_snapshot(snap, new, "vm_snap_running") + + @attr(tags=[TAG], required_hardware=True) + def test_16_vm_snapshot_create_stopped(self): + """ + VMS-03: VM snapshot of the stopped VM. + Writes before-s2 while Running, then stops and snapshots. + The snapshot is kept for step 17. + Verifies: state Ready; VM stays Stopped; one new FlexVol snapshot. + """ + vm = self._require_snapshot_vm() + self.__class__.token_s2 = new_token("before-s2") + self._write_marker(self.__class__.token_s2, disks=("ROOT", "DATA")) + self._stop_vm(vm.id) + before = self._ontap_snap_names() + snap = self._take_vm_snapshot(vm.id) + self.assertEqual(self._vm_state(vm.id), "Stopped") + new = self._assert_new_ontap_snaps(before, [snap], "vm", "VM snapshot (Stopped)") + self._keep_vm_snapshot(snap, new, "vm_snap_stopped") + + @attr(tags=[TAG], required_hardware=True) + def test_17_vm_snapshot_s1_running_s2_stopped_revert(self): + """ + VMS-09 (NFS ROOT + NFS DATA on the same pool): revert to the running + snapshot from step 15, then to the stopped snapshot from step 16. + Verifies: + - After revert to S1: ROOT and DATA hold before-s1 + - After revert to S2: ROOT and DATA hold before-s2 + - Both FlexVol snapshots survive the reverts + - Both VM snapshots are then deleted while the VM is still running + """ + if self.__class__.vm is None: + self.skipTest("VM not deployed — test_03 was skipped (no ready template)") + if self.__class__.vm_ssh is None: + self.skipTest("Guest SSH not set up — test_09 must pass first") + s1, s2, token_a, token_b, ontap_names = self._adopt_kept_vm_snapshots() + vm = self.__class__.vm + + self._stop_vm(vm.id) + VmSnapshot.revertToSnapshot(self.apiClient, s1.id) + self._start_vm(vm.id) + self._assert_marker(token_a, disk="ROOT", msg="after revert to S1") + self._assert_marker(token_a, disk="DATA", msg="after revert to S1") + self.assertEqual(len(self._vm_snapshots_of(vm.id)), 2, "S2 should still be listed") + + self._stop_vm(vm.id) + VmSnapshot.revertToSnapshot(self.apiClient, s2.id) + self._start_vm(vm.id) + self._assert_marker(token_b, disk="ROOT", msg="after revert to S2") + self._assert_marker(token_b, disk="DATA", msg="after revert to S2") + self._assert_ontap_snaps_present(ontap_names, "after both reverts") + + self._delete_vm_snapshot_while_running( + vm, s1, self._ontap_names_for_snap(ontap_names, s1, "vm"), token_b) + self._delete_vm_snapshot_while_running( + vm, s2, self._ontap_names_for_snap(ontap_names, s2, "vm"), token_b) + + @attr(tags=[TAG], required_hardware=True) + def test_18_vm_snapshot_duplicate_requests(self): + """ + VMS-53: two VM snapshot requests for the same VM at the same time. + A rejected request is retried on its own; one that already succeeded + is not repeated. + Verifies: both succeed and are Ready; two new FlexVol snapshots. + """ + vm = self._require_snapshot_vm() + before = self._ontap_snap_names() + pending = [ + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_dup_a_%d" % random.randint(0, 99999)), + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_dup_b_%d" % random.randint(0, 99999)), + ] + ok = [] + last_errors = [] + attempts = 3 + for attempt in range(1, attempts + 1): + results = _run_concurrent(*pending) + last_errors = [] + still_pending = [] + retryable = True + for fn, (result, exc) in zip(pending, results): + if exc is None and result is not None: + ok.append(result) + self._vm_snapshot_ids.append(result.id) + continue + last_errors.append(exc) + if exc is None or not _is_rejected(exc): + retryable = False + still_pending.append(fn) + pending = still_pending + if not pending: + break + if not retryable or attempt == attempts: + break + logger.warning("duplicate VM snapshot rejected, retry %d/%d: %s", + attempt, attempts, last_errors) + time.sleep(15) + self.assertFalse(pending, + "Duplicate VM snapshot requests failed: %s" % last_errors) + self.assertEqual(len(ok), 2, "Expected both VM snapshots to succeed; " + "got %d" % len(ok)) + for snap in ok: + self.assertEqual(snap.state, "Ready") + self.assertEqual(snap.virtualmachineid, vm.id) + self._assert_new_ontap_snaps(before, ok, "vm", "duplicate VM snapshot requests") + + # ------------------------------------------------------------------ + # Steps 17-21 - Second VM on the same pool: isolation and concurrency + # ------------------------------------------------------------------ + + @attr(tags=[TAG], required_hardware=True) + def test_19_deploy_second_vm(self): + """ + Deploy a second VM whose ROOT disk is on the same ONTAP pool. + Verifies: VM Running; ROOT on the test pool; SSH works. + """ + self._require_snapshot_vm() + cmd = deployVirtualMachineAPI.deployVirtualMachineCmd() + cmd.zoneid = self.zone.id + cmd.templateid = self.__class__.template_id + cmd.serviceofferingid = self.__class__.service_offering_id + cmd.account = self.account.name + cmd.domainid = self.domain.id + cmd.networkids = self.__class__.network_id + vm2 = self.apiClient.deployVirtualMachine(cmd) + self.__class__.vm2 = vm2 + self._poll_vm_state(vm2.id, "Running", timeout=900) + root = self._root_volume(vm2.id) + self.assertEqual(getattr(root, "storageid", None), self.__class__.pool.id, + "Second VM ROOT is not on the ONTAP test pool") + host, port, _ = open_ssh_port(self.apiClient, vm2.id, self.__class__.network_id) + self.__class__.vm2_ssh = (host, port) + token = new_token("second-vm-ssh-access-test") + self._write_marker(token, which="vm2") + self._assert_marker(token, which="vm2", msg="second VM access") + + @attr(tags=[TAG], required_hardware=True) + def test_20_volume_revert_leaves_other_vm_untouched(self): + """ + VS-28: two VMs on one FlexVol; revert VM-A's ROOT to its snapshot. + Verifies: VM-A is back at its snapshot; VM-B keeps data written after it. + """ + vm = self._require_snapshot_vm() + self._require_second_vm() + a1 = new_token("vm-a-before-snapshot") + a2 = new_token("vm-a-after-snapshot") + b1 = new_token("vm-b-before-snapshot") + b2 = new_token("vm-b-after-snapshot") + self._write_marker(a1) + self._write_marker(b1, which="vm2") + snap = self._take_volume_snapshot(self._root_volume(vm.id).id) + self._write_marker(a2) + self._write_marker(b2, which="vm2") + + self._stop_vm(vm.id) + Volume.revertToSnapshot(self.apiClient, snap.id) + self._start_vm(vm.id) + self._assert_marker(a1, msg="VM-A after revert") + self._assert_marker(b2, which="vm2", msg="VM-B after VM-A's revert") + + @attr(tags=[TAG], required_hardware=True) + def test_21_concurrent_vm_snapshots_two_vms(self): + """ + VMS-54: VM snapshots on two VMs of the same NFS pool at the same time. + Verifies: both Ready and owned by their VMs; two distinct new FlexVol + snapshots. + """ + vm = self._require_snapshot_vm() + vm2 = self._require_second_vm() + before = self._ontap_snap_names() + results = _run_concurrent( + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_cc_a_%d" % random.randint(0, 99999)), + lambda: VmSnapshot.create(self.apiClient, vm2.id, snapshotmemory="false", + name="ontap_cc_b_%d" % random.randint(0, 99999)), + ) + self._vm_snapshot_ids.extend(s.id for s, _ in results if s is not None) + errors = [e for _, e in results if e is not None] + self.assertFalse(errors, "Concurrent VM snapshots failed: %s" % errors) + snap_a, snap_b = results[0][0], results[1][0] + self.assertEqual(snap_a.state, "Ready") + self.assertEqual(snap_b.state, "Ready") + self.assertEqual(snap_a.virtualmachineid, vm.id) + self.assertEqual(snap_b.virtualmachineid, vm2.id) + self._assert_new_ontap_snaps(before, [snap_a, snap_b], "vm", "concurrent VM snapshots") + + @attr(tags=[TAG], required_hardware=True) + def test_22_concurrent_volume_snapshot_and_revert(self): + """ + Volume snapshot of VM-A's ROOT while VM-B's ROOT reverts, at the same time. + A rejected snapshot or revert is retried on its own. + Verifies: VM-A snapshot BackedUp with its own FlexVol snapshot; VM-B is + back at its snapshot; VM-A data unchanged. + """ + vm = self._require_snapshot_vm() + vm2 = self._require_second_vm() + a1 = new_token("vm-a-current-data") + b1 = new_token("vm-b-before-snapshot") + b2 = new_token("vm-b-after-snapshot") + self._write_marker(a1) + self._write_marker(b1, which="vm2") + snap_b = self._take_volume_snapshot(self._root_volume(vm2.id).id) + self._write_marker(b2, which="vm2") + self._stop_vm(vm2.id) + before = self._ontap_snap_names() + root_a = self._root_volume(vm.id).id + + snap_a = _snapshot_and_revert_with_retry( + lambda: Snapshot.create(self.apiClient, root_a, + account=self.account.name, domainid=self.domain.id), + lambda: Volume.revertToSnapshot(self.apiClient, snap_b.id), + ) + self._volume_snapshot_ids.append(snap_a.id) + self.assertEqual(snap_a.state, "BackedUp") + self._assert_new_ontap_snaps(before, [snap_a], "volume", "VM-A snapshot during VM-B revert") + + self._start_vm(vm2.id) + self._assert_marker(b1, which="vm2", msg="VM-B after revert") + self._assert_marker(a1, msg="VM-A during VM-B revert") + + @attr(tags=[TAG], required_hardware=True) + def test_23_concurrent_vm_snapshot_and_revert(self): + """ + VM snapshot of VM-A while VM-B reverts to its VM snapshot, at the same time. + A rejected snapshot or revert is retried on its own. + Verifies: VM-A snapshot Ready with its own FlexVol snapshot; VM-B is + back at its snapshot; VM-A data unchanged. + """ + vm = self._require_snapshot_vm() + vm2 = self._require_second_vm() + a1 = new_token("vm-a-current-data") + b1 = new_token("vm-b-before-snapshot") + b2 = new_token("vm-b-after-snapshot") + self._write_marker(a1) + self._write_marker(b1, which="vm2") + snap_b = self._take_vm_snapshot(vm2.id) + self._write_marker(b2, which="vm2") + self._stop_vm(vm2.id) + before = self._ontap_snap_names() + + snap_a = _snapshot_and_revert_with_retry( + lambda: VmSnapshot.create(self.apiClient, vm.id, snapshotmemory="false", + name="ontap_ccr_%d" % random.randint(0, 99999)), + lambda: VmSnapshot.revertToSnapshot(self.apiClient, snap_b.id), + ) + self._vm_snapshot_ids.append(snap_a.id) + self.assertEqual(snap_a.state, "Ready") + self._assert_new_ontap_snaps(before, [snap_a], "vm", "VM-A snapshot during VM-B revert") + + self._start_vm(vm2.id) + self._assert_marker(b1, which="vm2", msg="VM-B after revert") + self._assert_marker(a1, msg="VM-A during VM-B revert") + + # ------------------------------------------------------------------ + # Step 24 - Detach ONTAP data volume from the VM # ------------------------------------------------------------------ @attr(tags=["vm_volume_workflow"], required_hardware=True) - def test_09_detach_volume_from_vm(self): + def test_24_detach_volume_from_vm(self): """ Detach the ONTAP data volume from the running VM. Verifies: @@ -958,11 +1751,13 @@ def test_09_detach_volume_from_vm(self): ) # ------------------------------------------------------------------ - # Step 10 - Grow detached data volume + + # ------------------------------------------------------------------ + # Step 25 - Grow detached data volume # ------------------------------------------------------------------ @attr(tags=["vm_volume_workflow"], required_hardware=True) - def test_10_grow_detached_volume(self): + def test_25_grow_detached_volume(self): """Grow the materialized qcow2 after it has been detached.""" self.assertIsNotNone(self.__class__.volume, "Volume absent — test_02 must pass first") @@ -983,11 +1778,43 @@ def test_10_grow_detached_volume(self): self.__class__.target_volume_size = detached_target # ------------------------------------------------------------------ - # Step 11 - Destroy VM, delete volume, delete pool + # Step 26 - Destroy the second VM while it still has a VM snapshot # ------------------------------------------------------------------ @attr(tags=["vm_volume_workflow"], required_hardware=True) - def test_11_destroy_vm_and_cleanup(self): + def test_26_destroy_vm_with_vm_snapshot(self): + """ + VMS-38: destroy (expunge) the second VM while it still has a VM snapshot. + Verifies: destroy succeeds; the VM snapshot is no longer listed; its + FlexVol snapshot is removed from ONTAP. + """ + self._require_snapshot_vm() + vm2 = self._require_second_vm() + before = self._ontap_snap_names() + snap = self._take_vm_snapshot(vm2.id) + new = self._assert_new_ontap_snaps(before, [snap], "vm", "second VM snapshot") + + cmd = destroyVirtualMachineAPI.destroyVirtualMachineCmd() + cmd.id = vm2.id + cmd.expunge = True + self.apiClient.destroyVirtualMachine(cmd) + self.__class__.vm2 = None + self.__class__.vm2_ssh = None + self._vm_snapshot_ids.remove(snap.id) + + deadline = time.time() + 300 + while time.time() < deadline and self._vm_snapshots_of(vm2.id): + time.sleep(10) + self.assertFalse(self._vm_snapshots_of(vm2.id), + "VM snapshot still listed after destroying its VM") + self._wait_ontap_snaps_gone(new) + + # ------------------------------------------------------------------ + # Step 27 - Destroy VM, delete volume, delete pool + # ------------------------------------------------------------------ + + @attr(tags=["vm_volume_workflow"], required_hardware=True) + def test_27_destroy_vm_and_cleanup(self): """ Destroy the VM, delete the ONTAP data volume, enter maintenance, then force-delete the pool. @@ -1004,6 +1831,7 @@ def test_11_destroy_vm_and_cleanup(self): self.assertIsNotNone(self.__class__.pool, "Pool absent — test_01 must pass first") + self._destroy_second_vm() vm = self.__class__.vm vol = self.__class__.volume pool = self.__class__.pool @@ -1102,6 +1930,83 @@ def _list_vols_cmd(vol_id): return cmd +def _run_concurrent(*callables): + """Start all callables at the same moment; return [(result, exception)].""" + results = [(None, None)] * len(callables) + barrier = threading.Barrier(len(callables)) + + def worker(index, fn): + try: + barrier.wait(timeout=60) + results[index] = (fn(), None) + except Exception as exc: + results[index] = (None, exc) + + threads = [threading.Thread(target=worker, args=(i, fn)) + for i, fn in enumerate(callables)] + for thread in threads: + thread.start() + for thread in threads: + thread.join() + return results + + +def _is_rejected(exc): + """True when CloudStack or ONTAP refused the call because another one was in progress.""" + text = str(exc).lower() + return any(token in text for token in ( + "internal error", + "failed to create snapshot", + "failed to create instance snapshot", + "failed to revert", + "another operation", + "other active", + "try again later", + "in progress", + "busy", + )) + + +def _snapshot_and_revert_with_retry(snapshot_fn, revert_fn, attempts=3, pause=15): + """Run a snapshot and a revert together. + + A call that is rejected is retried on its own. A call that already + succeeded is not repeated. Returns the snapshot from snapshot_fn. + """ + snap = None + reverted = False + last_errors = [] + for attempt in range(1, attempts + 1): + jobs = [] + if snap is None: + jobs.append(("snapshot", snapshot_fn)) + if not reverted: + jobs.append(("revert", revert_fn)) + results = _run_concurrent(*(fn for _, fn in jobs)) + last_errors = [] + retryable = True + for (kind, _), (result, exc) in zip(jobs, results): + if exc is None: + if kind == "snapshot": + snap = result + else: + reverted = True + continue + last_errors.append("%s: %s" % (kind, exc)) + if not _is_rejected(exc): + retryable = False + if snap is not None and reverted: + return snap + if not retryable or attempt == attempts: + break + logger.warning("snapshot/revert rejected, retry %d/%d: %s", + attempt, attempts, last_errors) + time.sleep(pause) + raise AssertionError( + "Snapshot-while-revert failed after %d attempt(s): %s" + % (attempt, last_errors)) + + def _wait_for_vm_state(api_client, vm_id, target_state, timeout=120, interval=5): """Blocking wait for a VM to reach target_state (used in tearDownClass).""" diff --git a/test/integration/plugins/ontap/ontap.cfg b/test/integration/plugins/ontap/ontap.cfg index 64ee5174d379..47879ae4d340 100644 --- a/test/integration/plugins/ontap/ontap.cfg +++ b/test/integration/plugins/ontap/ontap.cfg @@ -121,6 +121,8 @@ "clusterName": null, "domainName": "ROOT", "templateName": "CentOS 5.5(64-bit) no GUI (KVM)", + "templateUsername": "root", + "templatePassword": "<>", "systemVmTimeoutSec": 3600, "templateReadyTimeoutSec": 3600, "pollIntervalSec": 60 diff --git a/test/integration/plugins/ontap/ontap_test_base.py b/test/integration/plugins/ontap/ontap_test_base.py index 37431dfa565e..e528aba2b25f 100644 --- a/test/integration/plugins/ontap/ontap_test_base.py +++ b/test/integration/plugins/ontap/ontap_test_base.py @@ -741,6 +741,27 @@ def delete_file_in_volume(self, vol_name, file_path): raise return True + def list_snapshots(self, vol_name): + """Return FlexVol snapshot records (name, uuid) for the named volume. + + Uses ``GET /api/storage/volumes/{uuid}/snapshots``. An empty list means + the volume does not exist or has no snapshots. + """ + vol = self.get_volume(vol_name) + if not vol: + return [] + vol_uuid = vol.get("uuid", "") + if not vol_uuid: + return [] + try: + data = self._get( + "/storage/volumes/%s/snapshots" % vol_uuid, + params={"fields": "name,uuid", "max_records": "1000"}, + ) + except Exception: + return [] + return list(data.get("records", []) or []) + # Invalid createStoragePool capacitybytes values. # (label, value, alternative fragment groups; any one group may match) diff --git a/test/integration/plugins/ontap/requirements.txt b/test/integration/plugins/ontap/requirements.txt new file mode 100644 index 000000000000..50c150856aa0 --- /dev/null +++ b/test/integration/plugins/ontap/requirements.txt @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Pinned dependencies for the ONTAP Marvin venv. +# Marvin itself is installed separately, from MARVIN_ARCHIVE or tools/marvin. +# setuptools<81 keeps pkg_resources, which nose needs to find the Marvin plugin. +# pyvmomi 9.x needs Python 3.10+. +# paramiko 4.x dropped SHA-1 key exchange, which the CentOS 5.5 guest sshd requires. +setuptools>=40.3.0,<81 +pyvmomi==8.0.2.0.1 +paramiko>=3.4,<4 diff --git a/test/integration/plugins/ontap/setup_env.sh b/test/integration/plugins/ontap/setup_env.sh index 2200e7c1c6f0..c0238a1ac1e8 100755 --- a/test/integration/plugins/ontap/setup_env.sh +++ b/test/integration/plugins/ontap/setup_env.sh @@ -54,11 +54,10 @@ if ! "${PIP}" install --upgrade pip wheel; then exit 1 fi -# nose discovers Marvin via setuptools entry points (needs pkg_resources). -if ! "${PIP}" install "setuptools>=40.3.0,<81"; then - echo "Unable to install a Marvin-compatible setuptools version" >&2 - exit 1 -fi +# requirements.txt is applied after Marvin. Marvin asks for +# setuptools>=40.3.0, paramiko>=1.13.0 and pyvmomi>=5.5.0, so its install +# would otherwise take the newest of each. +REQ="${ONTAP_DIR}/requirements.txt" if [[ -n "${MARVIN_ARCHIVE}" ]]; then if [[ ! -f "${MARVIN_ARCHIVE}" ]]; then @@ -89,9 +88,9 @@ else fi fi -echo "==> Pinning pyvmomi for Python 3.9 compatibility (pyvmomi 9.x requires 3.10+)" -if ! "${PIP}" install "pyvmomi==8.0.2.0.1"; then - echo "Unable to install pyvmomi; verify access to the configured Python package index" >&2 +echo "==> Installing pinned Python dependencies from requirements.txt" +if ! "${PIP}" install -r "${REQ}"; then + echo "Unable to install pinned Python dependencies; verify access to the configured Python package index" >&2 exit 1 fi @@ -125,7 +124,8 @@ fi echo "Nose test discovery compatibility OK" echo "" -echo "Done. Activate the venv with:" +echo "Done. Guest SSH uses paramiko>=3.4,<4 (CentOS 5.5 sshd is SHA-1 only)." +echo "Activate the venv with:" echo " source test/integration/plugins/ontap/.venv/bin/activate" echo "" echo "Run zone setup tests:"