Skip to content

Let CI test and check jobs use another runner pool, set by repository variables - #706

Merged
MaxGhenis merged 1 commit into
mainfrom
ci/runner-variables
Oct 24, 2026
Merged

MaxGhenis merged 1 commit into
mainfrom
ci/runner-variables

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Fixes #705

Draft: do not merge before Max decides on CI runner capacity (a cos decision; analysis in ~/reviews/ci-runner-capacity-2026-10/REPORT.md on the hub host). Companion: PolicyEngine/policyengine-uk#2598.

What and why

The org's 60-job limit for standard GitHub-hosted runners is shared by every PolicyEngine repository. Pushes to main, releases and the UK certification's qualify (up to 330 minutes, #660) and replay jobs queue for runners behind other repositories' pull request runs. This PR lets those jobs move to another runner pool, such as GitHub-hosted larger runners, by setting a repository variable, and move back by deleting it. Nothing changes until a variable is set.

Variable Jobs
CI_RUNNER_X64 pr_code_changes.yaml check-changelog, Lint, Mypy, Python-Compat, BundleVerification, Test; pr_docs_changes.yaml Test; push.yaml Lint, Test
CERTIFY_RUNNER_X64 certify-uk-release.yaml qualify, replay. These hold the GitHub App key and a write token, so point this only at a GitHub-hosted larger runner.
CI_RUNNER_EVENTS Optional: event names separated by commas, no spaces, for example push,workflow_dispatch. Only those events use the variables. Empty means every event.

Never moved (the test pins these): push.yaml Docs (Pages deploy), Versioning, Publish (PyPI trusted publishing), NotifyConsumers; certify-uk-release.yaml resolve, prepare, catch-up, accept, promote.

Read-only tokens. The org's default GITHUB_TOKEN permission is write (gh api orgs/PolicyEngine/actions/permissions/workflow). The pull request jobs had no permissions block, and push.yaml grants contents: write, pages: write and id-token: write at the workflow level, so Lint and Test on main ran with a write token. Every job CI_RUNNER_X64 can move now has permissions: contents: read. None of them uses the token beyond checkout.

Data. Test and BundleVerification read the dataset with HUGGING_FACE_TOKEN. The workflow comment says to point the variables only at runners where that data may be processed, such as the organisation's GitHub-hosted larger runners.

Each job's runner is

${{ vars.CI_RUNNER_X64 != '' && (vars.CI_RUNNER_EVENTS == '' || contains(format(',{0},', vars.CI_RUNNER_EVENTS), format(',{0},', github.event_name))) && vars.CI_RUNNER_X64 || 'ubuntu-latest' }}

An unset variable reads as '', so ubuntu-latest is used. The commas around both sides make pull_request not match pull_request_target.

To switch, for example to a larger runner named pe-x64-4core: Settings > Secrets and variables > Actions > Variables: CERTIFY_RUNNER_X64=pe-x64-4core (and CI_RUNNER_X64 if wanted). To switch back: delete the variable.

Invariants

  1. With no variable set, every job's runner is ubuntu-latest, as today.
  2. Only the 11 listed jobs can move; every other job keeps ubuntu-latest.
  3. A variable applies exactly when it is set and CI_RUNNER_EVENTS is empty or names the run's event as a whole name.
  4. Every job CI_RUNNER_X64 can move holds no write scope on GITHUB_TOKEN (id-token aside).

Test plan

  • tests/test_ci_runner_variables.py: invariants 1, 2 and 4 against the parsed workflows; invariant 3 exhaustively over 2 variable states × 6 event lists × 6 events (72 cases) on a Python mirror of the expression. 75 passed locally.
  • make lint (ruff format check and ruff check): clean.
  • actionlint on the changed workflows: no findings in this PR's lines. It reports one existing finding on main in push.yaml's Docs job (actions/configure-pages@v3 is too old), which this PR does not touch.
  • CI on this PR (variables unset, so it runs on ubuntu-latest).
  • Changelog fragment changelog.d/705.changed.md.

axiom: n/a: CI infrastructure only

🤖 Generated with Claude Code

The pull request checks, the docs build, and Lint and Test on pushes to main
read their runner from CI_RUNNER_X64; the UK certification's qualify and
replay jobs read CERTIFY_RUNNER_X64 (GitHub-hosted larger runners only: they
hold the App key and a write token). CI_RUNNER_EVENTS optionally limits the
switch to some events. ubuntu-latest stays the default, so nothing changes
until a variable is set. The jobs CI_RUNNER_X64 can move now hold a
read-only GITHUB_TOKEN instead of the organisation's default write token.
tests/test_ci_runner_variables.py holds the workflows to this.

Fixes #705

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 24, 2026 00:45
@MaxGhenis
MaxGhenis merged commit 8fe1522 into main Oct 24, 2026
13 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged on Max's go ("Merge policyengine-uk#2598 and policyengine.py#706 on their normal gates", 2026-10-24).

  • Head merged: 56517d7 (squash, --match-head-commit).
  • Checks: 13 of 13 pass at that head; gh pr checks exited 0 right before the merge.
  • Independent review: two Opus 5.5 reviews of the runner-capacity work covered this diff (round 1: no finding against this PR's code; its runner-events note was a report fix. Round 2: approve with fixes, all fixes in the report). The Codex lanes were closed, so there was no GPT-6.1 Sol pass.
  • Main had no workflow changes since the branch was cut.

Nothing changes until someone sets CI_RUNNER_X64 or CI_RUNNER_EVENTS on this repository; until then every job keeps its current runner. The read-only token on the moved jobs takes effect now.

@MaxGhenis
MaxGhenis deleted the ci/runner-variables branch October 24, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let CI test jobs use another runner pool, set by repository variables

1 participant