Repository navigation
Fix two kuttl test races: multi-pod log read and pgbouncer certificate rotation - #4570
Merged
ValClarkson merged 3 commits intoOct 16, 2026
Conversation
The replica-create backup Job is meant to fail, so Kubernetes replaces its Pod and several Pods carry the pgbackrest-backup=replica-create label by the time step 01 runs. `kubectl get pods -o name` returned all of them as one newline-joined string, and passing that to `kubectl logs` failed with "arguments in resource/name form may not have more than one slash". The log read came back empty and the step reported "did not find expected standby cluster error", which had nothing to do with standby clusters. Some of those Pods also exit on the pgBackRest configuration hash comparison before a backup runs, so their logs cannot contain the expected error even when nothing is wrong. Iterate over every matching Pod, retry the lookup and the log read together rather than reading once, and print each Pod's exit code and logs when the error is not found.
Step 12 wiped the stored PgBouncer certificate and then treated the first 'Loaded' line from the pgbouncer-config sidecar as proof that the regenerated certificate was live. That line can be the reload caused by the wipe itself: PGO regenerates the certificate within a couple of seconds, but kubelet needs considerably longer to propagate the updated Secret into the pod's projected volume. When the wipe's own reload was matched, step 13 read back the unchanged certificate and step 14 failed with 'Certificate did not change!'. Wait instead for the certificate mounted in the pod to differ from the one served before the wipe. That is the condition that decides what step 13 reads, so matching it removes the race rather than narrowing it.
kuttl runs a TestStep's commands exactly once and only loops the assert phase, so the retry helper in this file never retried; it printed a message, slept five seconds and failed the step. Every other file in the tree that uses that helper is a TestAssert. Make this one an assert too so kuttl owns the retrying, and drop the bounded shell loop that was standing in for it. Nesting a shell loop inside a command kuttl already re-runs multiplies the two timeouts and repeats the failure output on every attempt. The per-pod iteration stays, since a single pass still has to check every Pod. Move the verbose log dump into a collector, which kuttl runs once after the assert gives up rather than on each attempt.
crunchyheath
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two flaky kuttl e2e tests fix
pgbackrest-backup-standby — the Job is meant to fail, so Kubernetes retries
it and 4–5 labelled pods accumulate.
kubectl get pods -o namereturns one lineper pod, and passing that to
kubectl logs "${pod}"fails witharguments in resource/name form may not have more than one slash. The log variable ends upempty and the test misreports
did not find expected standby cluster error.Reproduces on an idle cluster.
pgbouncer — step 12 wipes the certificate, then accepts the first
Loadedline, which is usually the reload caused by its own wipe; step 14 then reports
Certificate did not change!. Failing run was 0.8s from patch toLoadedvs23–55s on passing runs, with
"frontend-tls.crt": no start linelogged 40msbefore the accepted line.
99853ecf4a8914d113b8186f4c1TestAssertso kuttl retries natively; diagnostics move tocollectors:00-assert.yamlis deliberately unchanged: both failure modes exit 1, becausethe Go wrapper ends at
log.Fatal(err), so they cannot be separated on exitcode. Commit 3 exists because
retry()was inert — kuttl runsTestStepcommands once and only loops the assert phase.