[europrinter (yankawai)]

What this PR does

adds an optional step to the v1.6 and next upgrade guides, right before helm upgrade. it pulls the new Cozystack images on the control plane nodes one node at a time, the etcd leader last, and checks etcd between nodes. the two steps after it are renumbered, nothing links to their anchors.

why, from cozystack/cozystack#4962. on Talos /var/lib/etcd and /var/lib/containerd are both directories on the EPHEMERAL volume (system_volumes.go), and Talos sets serializeImagePulls: false with no maxParallelImagePulls (kubelet_spec.go), so the rollout makes every node pull and unpack its new images at once. on a three-node cluster going from v1.6.4 to v1.6.5 the management etcd logged slow fdatasync up to 5.0 s and elected a new leader four times in five minutes, starting two minutes after helm upgrade. a later pull of one 370 MB image on the leader node gave 5.16 s and one more election, the same pull on a follower gave 1.07 and 1.31 s and no election.

on the second cluster we pulled the 25 changed images node by node before the upgrade, leader last, and the leader did not change during the pull or the rollout. the pull on the leader node still made it log 38 failed to send out heartbeat on time and one slow fdatasync of 1.2 s. during the rollout the leader logged none of them, one follower logged nine slow fdatasync. that cluster had two leader changes during its previous upgrade, v1.6.1 to v1.6.4, which ran without a pre-pull. we did not correlate those with pulls.

the leader is the member that matters. etcd runs the raft ticks and the WAL save in one loop (raft.go), so a leader stuck in fdatasync stops sending heartbeats and the followers start an election, the case the etcd FAQ puts on disk contention. a stalled follower does not tick either, so it does not campaign. that is why the guide pulls on the leader last and moves the leadership away with talosctl etcd forfeit-leadership before that pull.

options i looked at and left out

• maxParallelImagePulls in the Talos machine config spreads the pulls out, but on the first cluster one pull on the leader node was already enough for an election, and the cap slows every pull on the node. it needs its own testing.
• a dedicated ETCD volume removes the shared disk, but Talos accepts it from v1.14 and only when the node is created (v1.14.0 release notes, Dedicated System Volumes). Cozystack v1.6 ships Talos v1.13.6.
• shipping a pre-pull with the release, option 2 in the issue, needs node by node ordering that knows the etcd leader. not in this PR.

what i checked

• the image list snippet, copied from the page, run against a pod list built from the 1.6.4 images of the second cluster plus edge cases (an init container, an upstream image, a v1.6.40 tag, another registry, an older tag) gives the 26 expected refs with macOS, busybox and GNU grep and sed. without the (@|$) anchor or without the initContainers part it gives 27 and 25.
• all 24 ghcr.io/cozystack/cozystack/*:v1.6.5 tags from our list resolve to the digests pinned in v1.6.5 (sha256 of skopeo inspect --raw).
• etcd status, image pull --namespace cri and logs etcd are the talosctl commands we ran on the second cluster, Talos v1.13.6, and they exist in v1.14.4 too. forfeit-leadership moves the leadership to another member and does nothing on a follower (etcd.go). we did not run it together with a pull.
• Hugo 0.164.0 (ghcr.io/gohugoio/hugo:v0.164.0), npm ci, hack/download_openapi.sh, then the production build and the preview build with next/. both pass with the same warnings as main, and only the two upgrade pages differ from a main build.
• bash hack/check-i18n.sh check passes with 32 keys and 16 translated pages, hack/i18n/test_i18n.py runs 129 tests OK, this page has no translations. git diff --check is clean.

refs cozystack/cozystack#4962. it does not close the issue, option 2 is still open there.

[netlify]

✅ Deploy Preview for cozystack ready!

Name	Link
🔨 Latest commit	d9a1655
🔍 Latest deploy log	https://app.netlify.com/projects/cozystack/deploys/6ad8155631d6985b246f1156
😎 Deploy Preview	https://deploy-preview-762--cozystack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: fdae3a70-b7e9-44c6-b764-b910914d9aaf

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}