Skip to content

CI perf: yarn-classic 1.22.22 — one failed corepack yarn@1.22.22 fetch fails the leg and evicts a merge-queue entry (~30 MQ-min, ~94 job-min per hit) #1671

Closed

Description

Measurement

Window: 2026-10-21 20:16 → 2026-10-22 20:16 UTC. I fetched jobs for all 49 merge_group CI runs in that window.

Where the time goes

yarn-classic 1.22.22 p50 2.15 / p90 3.34 min (n=55). The time isn't spent in this job. It's spent on the eviction: the failed group, the green groups built on top of it that get rebuilt, and the PR's re-enqueue.

Root cause

require_yarn_classic (crates/socket-patch-cli/tests/common/yarn_classic_vex.rs:115) probes corepack yarn@<release> --version once, with no retry. scripts/yarn-classic-vex-matrix.sh sets SOCKET_PATCH_YARN_E2E_REQUIRED=1, which turns a failed probe into a panic. That's the right call for catching a missing yarn. The problem is that one transient fetch failure from corepack, on a host whose DNS is briefly timing out, becomes a hard test failure and evicts the entry from the merge queue.

Proposed fix

  1. In require_yarn_classic, retry the probe up to 3 times with backoff (2 s, 5 s) before calling skip. Panic only if every attempt fails, and print the last attempt's stderr, which the probe currently discards with Stdio::null().
  2. In scripts/yarn-classic-vex-matrix.sh, add one warm-up corepack yarn@$release --version into the tests' sandbox COREPACK_HOME before the suites run, with the same retry. Later probes then read the cached release instead of fetching it again.

The same pattern probably exists in the yarn-berry, pnpm and rush corepack probes. Check them while you're in there, but keep the PR small.

Expected saving

  • With 1 hit per ~50 merge-group runs (n=1 so far), and ~25 merges on a busy day, about 0.5 hits/day. That's ~15 merge-queue min/day amortized, on the queue's critical path, for the evicted PR and every green group behind it, plus about ~50 Linux job-min/day.
  • No change to PR feedback latency or macOS.

Coverage and risk

  • Every test still runs. A yarn release that really can't be fetched still fails after the retries, because REQUIRED=1 semantics are kept.
  • Worst case it adds ~7 s to a leg that is already failing.
  • Required checks ci-ok and clippy are unchanged.

Effort

S (one helper and one script line).

ROI

(0.05k job-min + 15 MQ-min) × confidence 0.4 (single observation) ÷ effort 1 ≈ 6.0

Activity

  1. mikolalysenko commented on Oct 22, 2026

    @mikolalysenko
    CollaboratorAuthor

    Working on this in the CI perf implementer.


    Generated by Claude Code

  2. mikolalysenko commented on Oct 22, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Triaged: p3 (CI-only). Open PR #1673 (ci-perf/1671-yarn-classic-probe-retry) addresses this, so leaving it open for that PR.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions