Repository navigation
Depot runners lost mid-job evicted 2 merge-queue entries in 27h #1660
Description
Two merge-queue entries were evicted in about 27 hours because a Depot runner (depot-ubuntu-24.04-4) stopped reporting partway through a job. GitHub then failed the job with "The self-hosted runner lost communication with the server". Both jobs normally finish in 1–5 minutes on the same label. The failed ones were declared lost about 10 minutes after the runner went quiet.
| When (UTC) | merge_group run | Job | Runner | Normal duration | Failed after |
|---|---|---|---|---|---|
| 2026-10-20 16:48 | 39460266425 (PR #1617) | e2e (ubuntu-latest, e2e_vex_build, pipenv) | depot-wkpbh4r80h | ~1 min | 10m00s |
| 2026-10-21 19:02 | 39657631762 (PR #1593) | e2e (ubuntu-latest, e2e_gradle_discovery_build …) | depot-v42fm5z0wd | ~5 min | 15m35s |
Both PRs merged after a rebuild, so each loss cost one full queue cycle (about 10 minutes plus the rebuild of every entry behind them). Neither job log could be downloaded, so I can't tell whether the runner died on its own or was starved for memory. The Gradle leg is the heaviest Linux e2e leg. The pipenv leg is light, which points away from memory pressure in that case.
This repo can't fix the cause; it's runner infrastructure. Next steps for a maintainer:
- Check the Depot dashboard for instance terminations or OOM kills at those times.
- If this keeps happening, consider larger runners for the e2e legs or a Depot support ticket.
The CI janitor will add new occurrences here.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 21, 2026
[agent] Triage (CI lane): this is runner infrastructure, so no change in this repo fixes it.
scripts/merge-queue-fail-fast.pyalready ignores runner-loss failures, which keeps a lost runner from cancelling sibling jobs. It cannot save the evicted entry, though. Labelled p3 and needs-human for the Depot dashboard and support-ticket checks above. The CI janitor keeps logging new occurrences here.Generated by Claude Code
mikolalysenko commentedon Oct 22, 2026
New occurrence (3rd eviction in ~49h), a different failure mode from the first two:
##[error]Step canceled by GitHub(Depot doc). Every other job in the run passed.The cancelled job failed
ci-ok, the queue removed #1664 at 17:24:11, and the five entries behind it (#1652, #1653, #1640, #1642, #1663) were rebuilt on new refs. That cost roughly one ~12 minute queue cycle for all of them. #1664 itself is still open and needs re-queueing once it's green.There's still no in-repo fix: the job never ran a step. Same next steps as above, plus checking the Depot dashboard for slow job assignment around 17:14–17:21 UTC on 2026-10-22.
Generated by Claude Code