Support numpy>=2 and remove numpy<2 pin

[kbvw]

No description provided.

[azure-pipelines]

✔️ 8904530 -> Azure artifacts URL

[azure-pipelines]

✔️ 5ba690e -> Azure artifacts URL

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.26%. Comparing base (c918f6b) to head (9fb93ec).
Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3040      +/-   ##
==========================================
- Coverage   67.27%   67.26%   -0.01%     
==========================================
  Files         571      571              
  Lines      104882   104887       +5     
==========================================
- Hits        70555    70550       -5     
- Misses      34327    34337      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[azure-pipelines]

✔️ 24e493b -> Azure artifacts URL

[azure-pipelines]

✔️ d30ad08 -> Azure artifacts URL

[azure-pipelines]

✔️ 16d874e -> Azure artifacts URL

[azure-pipelines]

✔️ 95b461c -> Azure artifacts URL

[azure-pipelines]

✔️ 7b9e1be -> Azure artifacts URL

[kbvw]

Wheels are now built with the latest Numpy and tested with the oldest Numpy that supports the corresponding Python version. A few notes:

1. When testing with different older Numpy versions, Python 3.8 kept failing over many runs of the CI. Since Python 3.8 is reaching its EOL very soon (and Numpy 2 does not support Python 3.8), I decided not to bother. It is still supported and tested, just with the same Numpy as it was built with (1.24 if I'm not mistaken) instead of the oldest Numpy that still supports Python 3.8.
2. For Windows, while debugging, it was straightforward to run these (quick) tests once with the Numpy it was built with, and then again with the oldest supported Numpy. I'm wondering if I should do the same for the Linux/Mac wheels, i.e. test before and after installing the older Numpy here. But the test_wheel function there does a lot of other tests that might take quite long and I'm not sure we want to do them twice for every wheel. What do you think? Alternatively, I can throw things around in that script to do neuron.test() and neuron.test_rxd() twice as in Windows, and do the other tests only once, either with latest or oldest Numpy.

[1uc]

might take quite long and I'm not sure we want to do them twice for every wheel

The "might take" is measurable. How long do they take? How long is that compared to building NEURON?

[mgeplf]

@kbvw > Python 3.8 kept failing over many runs of the CI.

Can you point me to a couple of those? I'm interested to see how it failed.

[kbvw]

@kbvw > Python 3.8 kept failing over many runs of the CI.

Can you point me to a couple of those? I'm interested to see how it failed.

They're a bit hard to find back because I force-pushed to the branch so often while debugging, but here is one example. This particular time, it was built with Numpy 1.24.4 and the RxD test fails with 1.20.3, with this:

'from neuron import rxd' failed numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

This was one in a range of runs where every time, it was built with 1.24.4 and then I tested with 1.17 (oldest Numpy that supports Python 3.8), 1.18, 1.19, etc. If I recall correctly it was still failing at 1.21, and as it was approaching the version it was built with, I decided I might as well stop. I did not investigate much, as Python 3.8 reaches EOL in October, and it's still built and tested, just with the same Numpy version it was built with. If it's important I can try to get to the bottom of it.

To be clear about what changed from before: we used to build the wheels with the oldest supported Numpy, and then test with the latest. As of Numpy 2, as you can read here, wheels built with 1.x will not run with 2.x, but wheels built with 2.x will run with an older 1.x. So in this PR, all the wheels are built with the latest Numpy and tested with the oldest supported Numpy. From Python 3.9, the latest Numpy is 2.x, and testing with the old 1.x versions seems to work fine. It is just Python 3.8 where we now have a reversed scenario from before. (It gets built with 1.24.4 instead of 1.17.3.)

[mgeplf]

@kbvw > 'from neuron import rxd' failed numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

Neat, thanks for the answer. That's what I was wondering. I remember running into similar problems in going from 1.19 to 1.20, since the size of ndarray changed.

I'm wondering if the build system is pickup various numpy headers instead of the ones that are specified?

As of Numpy 2, as you can read here, wheels built with 1.x will not run with 2.x, but wheels built with 2.x will run with an older 1.x.

That's very interesting. I wonder how they pulled that trick off, as even in the 1.xx days they couldn't guarantee backwards compat. The docs suggest it's for all the 1.xx series, but I wonder how they do that. Not important for this PR, just another thing I'd like to know.

[azure-pipelines]

✔️ b624267 -> Azure artifacts URL

[kbvw]

The "might take" is measurable. How long do they take? How long is that compared to building NEURON?

From the latest CI run, over the range of Linux and Mac Python wheels:

13m43s to 20m17s for the build script
3m49s to 4m44s for the test script
1m44s to 1m57s for the part that now repeats for two NumPy versions

So overal about a 9-11% increase.

Quickly testing with latest and oldest NumPy seems useful (especially instead of only oldest), but not sure if everything needs to be tested twice: MPI, nrnivmodl, etc. Maybe I'll re-arrange the script and just do neuron.test() and neuron.test_rxd() twice?

[azure-pipelines]

✔️ a665da1 -> Azure artifacts URL

[kbvw]

When running only neuron.test() and neuron.test_rxd() again with the oldest NumPy, this only takes about 4-6 seconds extra.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[azure-pipelines]

✔️ 9fb93ec -> Azure artifacts URL

[bbpbuildbot]

Logfiles from GitLab pipeline #228539 (:white_check_mark:) have been uploaded here!

Status and direct links:

• ✅ mac_m1_cmake_build: [cmake, ON, OFF, OFF, address]
• ✅ spack_setup
• ✅ build:nmodl
• ✅ build:neuron:nmodl:intel:legacy
• ✅ build:neuron:nmodl:intel:shared
• ✅ build:neuron:nmodl:nvhpc:acc:legacy
• ✅ build:neuron:nmodl:nvhpc:acc:shared
• ✅ build:neuron:nmodl:nvhpc:omp:legacy
• ✅ build:neuron:nmodl:nvhpc:omp
• ✅ test:neuron:nmodl:intel:legacy
• ✅ test:neuron:nmodl:intel:shared
• ✅ test:neuron:nmodl:nvhpc:acc:legacy
• ✅ test:neuron:nmodl:nvhpc:acc:shared
• ✅ test:neuron:nmodl:nvhpc:omp:legacy
• ✅ test:neuron:nmodl:nvhpc:omp

[1uc]

Nice work.