Skip to content

CI: retry the website deploy step on a transient Cloudflare failure, with a bounded window #970

Open

Description

@chappie-daemon

What happened

Run 40103130697 failed at Deploy the preview worker:

✘ [ERROR] A request to the Cloudflare API (/accounts/…/workers/scripts/nostream-web-pr-969/assets-upload-session) failed.
  An unknown error has occurred. [code: 10013]

The build had already succeeded and 164 assets were read. The run one minute earlier on the same branch passed, the failing commit only deleted five lines from Peering.astro, and the next push deployed successfully — so this is a transient upstream failure, not a defect in the site or the workflow.

One transient failure out of roughly five runs is enough that the fix is worth having, but only if it does not turn a visible outage into an invisible one.

What to build

A retry that wraps the deploy step — not the build — in both deploy-website-preview.yml and deploy-website.yml:

  • 3 attempts, backoff of about 10 s then 30 s so the total added time is bounded at roughly a minute.
  • A per-attempt timeout and a step-level timeout-minutes, so a hung upload cannot burn the job.
  • Retry only transient signals: code: 10013, an HTTP 5xx, or a connection reset. An authentication failure, a missing environment, or a wrangler validation error must fail on the first attempt — retrying those triples the noise and delays the real message.
  • A ::warning:: annotation whenever an attempt is not the first, naming the attempt number and the reason. Without this the retry hides the flake rate entirely, and we would never learn that Cloudflare is degrading or that a real limit is approaching.

Implementation notes

  • Both workflows deploy the same way (npx wrangler deploy --config … --name …), so the retry belongs in one shared script under .github/scripts/, invoked by both. Two copies will drift.
  • Prefer a shell loop over a third-party retry action: it keeps the supply chain as it is, and it makes the transient-signal filter ours to write and read.
  • The script should be testable the way the release tag step is — a test that executes the script against a stub wrangler that fails once and then succeeds, plus one that fails with a 401 and asserts there was no second attempt.

Out of scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions