Skip to content

Cache the JIGSAW build in CI - #1009

Draft
xylar wants to merge 7 commits into
MPAS-Dev:mainfrom
xylar:cache-jigsaw-in-ci
Draft

xylar wants to merge 7 commits into
MPAS-Dev:mainfrom
xylar:cache-jigsaw-in-ci

Conversation

@xylar

@xylar xylar commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

This PR caches mache's JIGSAW build in CI, so ./deploy.py reuses it instead of spending about three of a job's six minutes rebuilding. The build, docs and GIS workflows share one cache key, so pull requests restore the cache that main saves. The key changes with the jigsaw-python submodule, the Python version and the mache pin.

This is on top of #1008, which brings in mache 5.2.0 and its JIGSAW cache. Only the last commit is new here, and I will rebase once #1008 merges.

For the reviewer:

  • The key is defined once, in a composite action under .github/actions/, rather than repeated in each workflow the way the pixi cache step is.

Checklist

  • Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar and others added 7 commits October 23, 2026 08:01
This reverts commit 0a8c5da, which pointed CI at an old mache branch
whose jigsaw build pins libnetcdf 4.10.0, conflicting with the 4.10.1
pin in deploy/pins.cfg.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every CI job that runs ./deploy.py rebuilds JIGSAW, which takes about
three minutes. mache 5.2.0 already keeps a content-keyed build cache
under .mache_cache/jigsaw, so persist it with actions/cache in the
build, docs and GIS workflows.

A composite action holds the key, so the three workflows cannot drift
apart and miss the cache that main's build workflow saves. The key uses
the jigsaw-python submodule commit (which hashFiles cannot see), the
Python version and the mache pin. The mache pin is needed because
mache's key includes its own build recipe: without it, a primary-key
hit on a stale slot would rebuild on every run and never save.

The path lists the children of .mache_cache/jigsaw because
actions/cache globs without implicit descendants, so an exclusion
under a bare directory path would be ignored. This keeps the shared
build tools (about 110 MB) out of the cache.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar

xylar commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Testing

I re-ran CI on this branch after the first run saved the caches. Every job restored its cache and logged Using cached JIGSAW build:, and the "Install compass" step got about three minutes shorter:

Job First run Re-run
python 3.11 185 s 52 s
python 3.12 233 s 35 s
python 3.13 165 s 38 s
python 3.14 230 s 39 s
GIS mesh gen 253 s 60 s

All jobs passed. The GIS job needed a second re-run after mpirun failed in MPI_Init on one runner, after the cache had already been restored.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar added a commit to E3SM-Project/polaris that referenced this pull request Oct 23, 2026
Every CI job that runs ./deploy.py rebuilds JIGSAW, which takes about
three minutes. mache already keeps a content-keyed build cache under
.mache_cache/jigsaw, so persist it with actions/cache in the build and
docs workflows.

A composite action holds the key, so the workflows cannot drift apart
and miss the cache that main's build workflow saves. The key uses the
jigsaw-python submodule commit (which hashFiles cannot see), the Python
version and the mache pin. The mache pin is needed because mache's key
includes its own build recipe: without it, a primary-key hit on a stale
slot would rebuild on every run and never save.

The path lists the children of .mache_cache/jigsaw because
actions/cache globs without implicit descendants, so an exclusion
under a bare directory path would be ignored. This keeps the shared
build tools (about 110 MB) out of the cache.

This is the same change as MPAS-Dev/compass#1009.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar xylar added ci Changes affect Azure Pipelines CI clean-up labels Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Changes affect Azure Pipelines CI clean-up

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant