Skip to content

MAINT: locking with uv, remove requirements.txt #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattip merged 6 commits into from
Oct 14, 2026
Merged

MAINT: locking with uv, remove requirements.txt #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
73969bc
MAINT: locking with uv, remove requirements.txt
andyfaff Oct 12, 2026
554e2d7
MAINT: bump actions hashes
andyfaff Oct 13, 2026
ca97ca4
DOC: scipy-->numpy
andyfaff Oct 13, 2026
65f38fa
BLD: harden sdist
andyfaff Oct 13, 2026
d4305af
BLD: harden sdist
andyfaff Oct 13, 2026
d0ed775
BLD: build all
andyfaff Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 141 additions & 39 deletions .github/workflows/wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ env:
# to build wheels. This is normally set to `main` in the main branch of this
# repo, and to a tag name (e.g., `v2.3.2`) on a release branch.
SOURCE_REF_TO_BUILD: main
UV_VERSION: "0.12.18"
UV_HASH: "89eadd7c76fc063887959510d5ba0ab1264dfd5f1143b925ddb73021a40acf16"


on:
schedule:
Expand Down Expand Up @@ -38,21 +41,70 @@ permissions:
contents: read # to fetch code (actions/checkout)

jobs:
check_lock:
name: Check uv.lock file
runs-on: ubuntu-latest
if: (github.repository == 'numpy/numpy-release' || github.repository == '')

steps:
- name: Checkout numpy-release
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Checkout numpy
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: numpy/numpy
ref: ${{ env.SOURCE_REF_TO_BUILD }}
path: numpy-src
persist-credentials: false

- name: Install uv
uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
with:
version: ${{ env.UV_VERSION }}
# unlike build_wheels, this job is Linux-only, so we can pin the download too
checksum: ${{ env.UV_HASH }}
enable-cache: false

- name: Check lock file
run: |
# pyproject.toml holds the dependency groups the builds install, copied out
# of numpy's; regenerating it here is what catches numpy changing them.
# See CONTRIBUTING.md for how to regenerate these files. The cooldown the
# lock was generated with is recorded in it, and has to be repeated here.
python3 tools/sync_dependency_groups.py numpy-src | diff -u pyproject.toml -
uv lock --check --exclude-newer "7 days"

- name: Record the pins that will be used
run: |
# the packages that actually get installed, for review of a lock file bump
{
echo '## Build and test dependencies for this run'
echo '```'
uv export --frozen --no-hashes --no-emit-project --no-default-groups \
--group build --group openblas32 --group openblas64 --group test
echo '```'
} | tee -a "$GITHUB_STEP_SUMMARY"


build_wheels:
if: github.repository == 'numpy/numpy-release'
name: Build wheel ${{ matrix.python }}-${{ matrix.buildplat[1] }}-${{ matrix.buildplat[2] }}
name: BLD > ${{ matrix.python }}-${{ matrix.buildplat[1] }}-${{ matrix.buildplat[2] }}
runs-on: ${{ matrix.buildplat[0] }}
needs: [check_lock]
strategy:
# Ensure that a wheel builder finishes even if another fails
fail-fast: false
matrix:
# Github Actions doesn't support pairing matrix values together, let's improvise
# https://github.com/github/feedback/discussions/7835#discussioncomment-1769026
buildplat:
- [ubuntu-22.04, manylinux_x86_64, ""]
- [ubuntu-22.04, musllinux_x86_64, ""]
- [ubuntu-22.04-arm, manylinux_aarch64, ""]
- [ubuntu-22.04-arm, musllinux_aarch64, ""]
- [ubuntu-24.04, manylinux_x86_64, ""]
- [ubuntu-24.04, musllinux_x86_64, ""]
- [ubuntu-24.04-arm, manylinux_aarch64, ""]
- [ubuntu-24.04-arm, musllinux_aarch64, ""]
- [macos-15-intel, macosx_x86_64, openblas]

# targeting macos >= 14. Could probably build on macos-14, but it would be a cross-compile
Expand All @@ -67,7 +119,7 @@ jobs:
python: "cp314t"
include:
- buildplat: [ ubuntu-22.04, pyodide_wasm32, "" ]
python: "cp313"
python: "cp314"
Comment thread
mattip marked this conversation as resolved.


steps:
Expand All @@ -85,19 +137,35 @@ jobs:
submodules: true
persist-credentials: false

# the version of python we'll run cibuildwheel with
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: 3.13

# Used in cibw_before_build to install pinned build dependencies
# Note that cibuildwheel provides it in the Linux images, so this install
# is only useful on macOS/windows
- name: Install uv
uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
with:
# can't use checksum here, because there are different packages
# for different OS
version: ${{ env.UV_VERSION }}
enable-cache: false

- name: win_arm64 - set environment variables
if: ${{ matrix.buildplat[1] == 'win_arm64' }}
run: |
echo "C:\Program Files\LLVM\bin" >> $env:GIHUB_PATH
echo "CC=clang-cl" >> $env:GITHUB_ENV
echo "CXX=clang-cl" >> $env:GITHUB_ENV
echo "FC=flang" >> $env:GITHUB_ENV

- name: pkg-config-for-win
- name: Windows setup
if: runner.os == 'windows'
env:
GITHUB_WORKSPACE: ${{ github.workspace }}
run: |
rm -fo c:/Strawberry/c/lib/pkgconfig/*.pc
$CIBW = "$env:GITHUB_WORKSPACE/.openblas"
# pkgconfig needs a complete path, and not just "./openblas since the
# build is run in a tmp dir (?)
Expand All @@ -112,7 +180,7 @@ jobs:
BLAS_BACKEND: ${{ matrix.buildplat[2] }}
run: |
# Needed due to https://github.com/actions/runner-images/issues/3371
# Supported versions: https://github.com/actions/runner-images/blob/main/images/macos/macos-14-arm64-Readme.md
# Supported versions: https://github.com/actions/runner-images/blob/main/images/macos/macos-15-arm64-Readme.md
echo "FC=gfortran-15" >> "$GITHUB_ENV"
echo "F77=gfortran-15" >> "$GITHUB_ENV"
echo "F90=gfortran-15" >> "$GITHUB_ENV"
Expand All @@ -128,9 +196,10 @@ jobs:
if: ${{ matrix.buildplat[1] == 'pyodide_wasm32' }}
run: |
echo "CIBW_PLATFORM=pyodide" >> "$GITHUB_ENV"
echo "CIBW_ENVIRONMENT_PYODIDE=INSTALL_OPENBLAS=false" >> "$GITHUB_ENV"

- name: Build wheels
uses: pypa/cibuildwheel@1828c10ab37f080699c7b81cea34097c684a7074 # v4.2.0
uses: pypa/cibuildwheel@b3395e271a35f73dad77f21802abe9b986dfe299 # v4.3.1
env:
CIBW_BUILD: ${{ matrix.python }}-${{ matrix.buildplat[1] }}
SRC: ${{ github.workspace }}/numpy-src
Expand All @@ -139,7 +208,7 @@ jobs:
output-dir: dist
config-file: cibuildwheel.toml

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: ${{ matrix.python }}-${{ matrix.buildplat[1] }}-${{ matrix.buildplat[2] }}
path: ./dist/*.whl
Expand All @@ -154,57 +223,89 @@ jobs:

steps:
- name: Download wheels
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: dist
merge-multiple: true

- name: Install micromamba
uses: mamba-org/setup-micromamba@ce51e99f4bb8a82ab7158c4dc59ef4634c59c4f9 # v3.1.0
- name: Upload nightly wheel

@andyfaff andyfaff Oct 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Easier to use a bespoke action that's designed specifically to upload to nightly wheels.

uses: scientific-python/upload-nightly-action@16fa02eacee1655195143de09f03676e60ef2bf5 # 0.6.5
with:
# For installation of anaconda-client, required for upload to anaconda.org
init-shell: bash
environment-name: upload-env
create-args: >-
anaconda-client

- name: Upload to anaconda.org
shell: bash -el {0} # required for micromamba
env:
TOKEN: ${{ secrets.NUMPY_NIGHTLY_UPLOAD_TOKEN }}
run: |
anaconda -q -t ${TOKEN} upload --force -u scientific-python-nightly-wheels ./dist/*.whl

anaconda_nightly_upload_token: ${{ secrets.NUMPY_NIGHTLY_UPLOAD_TOKEN }}
artifacts_path: dist/

# -------------------------------------------------------------------------------------
build_sdist:
name: Build sdist
runs-on: ubuntu-latest
if: github.repository == 'numpy/numpy-release'
needs: [ check_lock ]
steps:
- name: Checkout numpy-release
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Checkout numpy
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: numpy/numpy
submodules: true
ref: ${{ env.SOURCE_REF_TO_BUILD }}
path: numpy-src
submodules: true
persist-credentials: false

- name: Install uv
uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
with:
version: ${{ env.UV_VERSION }}
# unlike build_wheels, this job is Linux-only, so we can pin the download too
checksum: ${{ env.UV_HASH }}
enable-cache: false

- name: Build sdist
run: |
python -m pip install -U pip build
python -m build --sdist -Csetup-args=-Dallow-noblas=true
# use uv and a lock file to pin down all dependencies for the sdist build.
# The lock file lives in this repo, so sync from here; the environment goes
# into the numpy checkout, where `uv build --no-build-isolation` looks for it.
# --frozen : uses specification in uv.lock as source of truth
# only-managed: force uv to use a managed python. No version is requested, so
# it's uv's default managed CPython - i.e. UV_VERSION decides which Python
# the sdist is built and tested with; cibuildwheel isn't used in this job.
export UV_PROJECT_ENVIRONMENT=$GITHUB_WORKSPACE/numpy-src/.venv
uv sync --python-preference only-managed --frozen --no-default-groups --group build

cd numpy-src
uv build --sdist --no-build-isolation -Csetup-args=-Dallow-noblas=true .

- name: Install the sdist
run: |
# the sdist build environment has done its job; removing it proves that the
# test environment doesn't quietly depend on it
rm -rf $GITHUB_WORKSPACE/numpy-src/.venv

cd numpy-src/dist

# build and install the sdist in a fresh venv, populated by the same
# `uv sync` from the same lock file as the build environment above
export UV_PROJECT_ENVIRONMENT=$GITHUB_WORKSPACE/numpy-src/dist/test
uv sync --project $GITHUB_WORKSPACE --python-preference only-managed --frozen --no-default-groups --group build --group test
source test/bin/activate
uv pip install --no-deps --no-build-isolation -v *.gz

- name: Test the sdist
run: |
python -m pip install dist/*.gz -Csetup-args=-Dallow-noblas=true
pip install -r requirements/test_requirements.txt
cd .. # Can't import numpy within numpy src directory
# each step gets a fresh shell, so re-enter the venv built above
cd numpy-src/dist
source test/bin/activate
python -c "import numpy, sys; print(numpy.__version__); sys.exit(numpy.test() is False)"

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: sdist
path: ./dist/*
path: numpy-src/dist/*.tar.gz

# -------------------------------------------------------------------------------------
testpypi-publish:
name: Publish release to TestPyPI
if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'testpypi'
Expand All @@ -217,7 +318,7 @@ jobs:
id-token: write # mandatory for trusted publishing
steps:
- name: Download sdist and wheels
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: dist
merge-multiple: true
Expand All @@ -230,13 +331,14 @@ jobs:
ls -al

- name: Publish
uses: pypa/gh-action-pypi-publish@ba38be9e461d3875417946c167d0b5f3d385a247 # v1.14.1
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
with:
repository-url: https://test.pypi.org/legacy/
skip-existing: true
print-hash: true
attestations: true

# -------------------------------------------------------------------------------------
check_version:
name: Ensure commit is tag before upload to PyPi

Expand Down Expand Up @@ -279,7 +381,7 @@ jobs:
id-token: write # mandatory for trusted publishing
steps:
- name: Download sdist and wheels
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: dist
merge-multiple: true
Expand All @@ -292,7 +394,7 @@ jobs:
ls -al

- name: Publish
uses: pypa/gh-action-pypi-publish@ba38be9e461d3875417946c167d0b5f3d385a247 # v1.14.1
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
with:
print-hash: true
attestations: true
57 changes: 57 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,63 @@ fixes are very welcome, however CI jobs will not run for anyone who doesn't
have commit access.


## Updating the pinned dependencies

Every build and test dependency is pinned, with hashes, in `uv.lock`. The dependency
groups it pins live in the committed `pyproject.toml`, which is *generated* from the
`build`, `test`, `openblas32`, `openblas64`, `test-core`, `hypothesis` groups from
the `pyproject.toml` of`numpy/numpy`. It needs to be consistent with the `numpy/numpy`
commit this branch builds (`SOURCE_REF_TO_BUILD` in`.github/workflows/wheels.yml`). The
build scripts install from the lock file with `uv export --require-hashes`.

Locking numpy's own `pyproject.toml` would work too, but it would pin all of its
dependency groups - `doc`, `dev`, `typecheck` and the rest - which is about 180 extra
packages that are never installed here, and a lock file six times the size.

Because `pyproject.toml` is a copy, it goes stale whenever numpy changes those three
groups. The `check_lock` CI job uses `tools/sync_dependency_groups.py` to
generate a `pyproject.toml` from `numpy/numpy`, diffs it against the committed one (in
this repository), and then runs `uv lock --check`, before any wheels are built.

To regenerate both `uv.lock` and `pyproject.toml`:

```bash
tools/update_lock.sh # sync to numpy's dependency groups
tools/update_lock.sh --upgrade # ... and also bump every pin to the latest release
```

This expects a `numpy/numpy` checkout at `../numpy`, at the commit named by
`SOURCE_REF_TO_BUILD`; set `$NUMPY_SRC` if yours lives somewhere else. Use a `uv` at
least as new as CI's `UV_VERSION` - older ones write a different lock file format.

Releases published within the last 7 days are ignored, so that a version published in the
last few days can't end up in a build. uv writes that window into the lock file and
`uv lock --check` only passes when given the *same* value, so it is hardcoded in two
places that have to agree: the `check_lock` job in `wheels.yml`, and
`tools/update_lock.sh`. Change one and CI will reject the lock file you generate.

## Reviewing a lock file change

Check `pyproject.toml` first: it determines everything else, and a change to it should
correspond to a change in numpy's dependency groups. `check_lock` proves that it does.

For `uv.lock` itself, worth checking on top of the version changes:

- That the diff contains no `source = { registry = ... }` pointing anywhere other than
`https://pypi.org/simple`, and no `source = { url = ... }` or `{ git = ... }` entries.
- That `requires-python` and the `[options]` block at the top of the file are unchanged;
a dropped `exclude-newer-span` means the cooldown was silently skipped.
- That the number of packages hasn't grown unexpectedly. The `check_lock` job writes the
packages that get installed to its job summary on every run, so a PR touching the lock
file can be reviewed by comparing its summary against the one from the most recent run
on `main`. To produce that list locally:

```bash
uv export --frozen --no-hashes --no-emit-project --no-default-groups \
--group build --group openblas32 --group test-core --group test --group openblas64
```


## Running CI jobs on your own fork

To get CI to run on your own fork for changes in a branch named
Expand Down
Loading
Loading