Skip to content

Flake: nightly poetry hosted rollback fails on one transient PyPI request (UpstreamClient has no retry) #1650

Open

Description

Nightly e2e-extended (ubuntu-latest, e2e_vex_build, poetry:: --ignored, 1.1.15) failed once on a single failed PyPI request during hosted rollback.

Evidence

  • Run https://github.com/SocketDev/socket-patch/actions/runs/39101574188 (schedule, main, 2026-10-18), job 117599202945
  • poetry::poetry_hosted_fresh_install_then_manifestless_vex panicked at crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs:898: rollback exited 1 with hosted_restore_refused:
    cannot restore pkg:pypi/six@1.16.0 to its upstream registry entry: six==1.16.0: GET https://pypi.org/pypi/six/1.16.0/json: error sending request for url (...)
  • The other 3 poetry tests in the same job reached PyPI fine seconds earlier. The 10-19 nightly passed on the same code.
  • Frequency: 1 of the last 12 nightlies. It has not been seen on PR or merge_group runs (this leg is nightly-only).

Root cause (partly known)
UpstreamClient (crates/socket-patch-core/src/patch/redirect/upstream/client.rs) sends each registry lookup once. get_json_authorized and download turn any transport error into a final GET {url}: {e} with no retry. The patch API client does retry dropped connections and 429/503 (api::retry, send_json_request). Because the message keeps only reqwest's top-level text, the log can't tell us the transport error class (DNS, reset, timeout).

Proposed fix (not started: the evidence was too thin this run)

  1. Format registry-lookup errors with the cause chain, the same way api::client::network_error_detail does, so the next occurrence is classifiable.
  2. If it recurs as a dropped connection or a 429/503, give UpstreamClient's GETs the same bounded ApiRetry policy that send_json_request uses (is_retryable_transport plus is_retryable_status, the shared retry window). A user's hosted rollback hits the same failure mode.

Filed by the hourly CI janitor; the job log expires, so the signature is quoted above.


Generated by Claude Code

Activity

  1. mikolalysenko commented on Oct 21, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Triaged (CI lane): p3, nightly-only flake (1 in 12 nightlies, not seen on PR or merge-queue runs), so it costs no PR/queue runner time. No open PR covers it yet. The proposed fix is in UpstreamClient (core), so it also affects users doing a hosted rollback. Not claimed this run because the CI-lane fixer already has 3 open PRs (#1641, #1642, #1647) waiting on review.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions