CI fails on Windows conda-forge OpenBLAS job #266

Activity

1. virchan commented on Oct 14, 2026

   

   virchan

   on Oct 14, 2026

   ContributorAuthor

   More actions

   The job crashed with an access violation: https://github.com/joblib/threadpoolctl/actions/runs/38332722465/job/115085934734#step:6:295

   Windows fatal exception: access violation

   The crash occurred here:

   threadpoolctl/tests/empirical_scope_observation.py

   Line 59 in 24d5d29

   A.dot(A)

   However, CI was green two days ago, and we don't have any new commits into master since then. So, I started to compare the environments.

   To identify the difference between environments, Claude suggested saving the CI logs separately and running diff. The only difference is the Python build (101 vs 102). In particular, libopenblas, numpy, scipy and the VC runtime are identical in both runs.

   diff green.txt red.txt

   122c121
   < cpython                                 3.14.8                 py314hd8ed1ab_101      conda-forge
   ---
   > cpython                                 3.14.8                 py314hd8ed1ab_102      conda-forge
   196c195
   < libpython                               3.14.8                 h4f90d01_101_cp314     conda-forge
   ---
   > libpython                               3.14.8                 hc6187dd_102_cp314     conda-forge
   231,232c230,231
   < python                                  3.14.8                 h53f6dd8_101_cp314     conda-forge
   < python-gil                              3.14.8                 h4df99d1_101           conda-forge
   ---
   > python                                  3.14.8                 h53f6dd8_102_cp314     conda-forge
   > python-gil                              3.14.8                 h4df99d1_102           conda-forge

   To determine whether the Python 101 -> 102 rebuild is a cause, I performed an A/B test with two PRs:

   • [DO NOT MERGE] trigger CI run for python build 102 #267 runs the same CI as in master.
   • [DO NOT MERGE] trigger CI run for python build 101 #268 pins the Python build to 101 in the Windows OpenBLAS job.

   The Windows OpenBLAS job passed in both PRs. In particular, #267 ran with the same Python 102 build, the same libopenblas (0.3.34 pthreads_h952cde7_2), the same numpy and the same Zen architecture as the failing run, so the rebuild alone does not trigger the crash.

   I would suspect there was a race, but I couldn't confirm it. Also, re-running the CI would probably fix this.
2. lesteve commented on Oct 14, 2026

   

   lesteve

   on Oct 14, 2026

   Last edited by lesteve

   Member

   More actions

   Thanks a lot @virchan for having a look 🙏 ! I definitely appreciate any kind of help on looking at weird CI issues 😉.

   This seems similar to #261 right? If yes the tl;dr is that it may be already fixed in the OpenBLAS develop branch. See #261 (comment) for more details.

   👍React with 👍1Reacted by Virgil Chan
3. virchan commented on Oct 15, 2026

   

   virchan

   on Oct 15, 2026

   ContributorAuthor

   More actions

   This seems similar to #261 right?

   Indeed! Thank you for reference!

   Closing this issue to keep the discussion in one place #261.
4. virchan

   closed this as completedon Oct 15, 2026
5. virchan

   closed this as a duplicateof CI Intermittent Windows fatal exception: access violation in tests/empirical_scope_observation.py #261 on Oct 15, 2026