Dealing with NumPy build time requirements for the 2.6.x release

[IAlibay]

Some background

As detailed in #4183 the NumPy C API became backwards compatible as part of NumPy 1.25+. Prior to that point, we used to build with the "oldest supported NumPy version" (currently 1.22.3), which guaranteed forward compatibility.

The issue

In 2.6.0 (specifically #4198) we switch out our pyproject.toml build section to just use numpy >=1.25. This worked great for pip installs.

However what we hadn't accounted for is the fact that conda-forge does not currently have the adequate amount of machinery to handle this.

Everything in conda-forge is set up to deal with a build where you use an "oldest supported NumPy version", so if you try to build with v1.25, you end up with an install pin range of "numpy >=1.25,<2.0". This is problematic since we need to be able to support old versions of NumPy and conda-forge is the most popular way folks install MDAnalysis.

The status of conda-forge support & how other packages are doing it

My understanding is that this won't be fully resolved until the 2.0 release of NumPy (see: conda-forge/conda-forge-pinning-feedstock#4816).

Indeed scipy eventually decided to punt the issue to at least the next minor release.

The solutions

There are two possible solution here:

1. We revert the change introduced in 2.6.0, cherry pick that change and put out a 2.6.1 release. This would ensure that there is no discrepancies in the build-time NumPy versions between our deployments and also that we don't have to deal with further headaches here until after NumPy and SciPy lead the way for us.
2. We ignore the difference between PyPi and conda-forge builds, build with new numpy in one and old numpy in another.

Which one would create less work

A priori option number two would seem like less work, but honestly I think number one would allow us to bury our heads in the sand for a bit longer - also we've just done a release, so we know all the machinery works right now. It would be less painful to do a bugfix release now than to do it later.

My personal preference is just to do option one and be done with it.

[IAlibay]

@MDAnalysis/coredevs can we come to a quick decision here please? I'd like to get this done with by middle of the week (after that is the NF conference and then the UGM, so time will be too restricted).

I'm going to set up the necessary changes now for option one so I can just cherry pick a release asap.

[richardjgowers]

I don't think we need to have our np pins identical between the two, so option 2 is fine? But if you don't feel option 1 is a lot of work that's fine too.

[lilyminium]

Option 1 seems like the better option generally, so if you're happy to do option 1 I'm happy with that. Otherwise option 2 works too.

[IAlibay]

But if you don't feel option 1 is a lot of work that's fine too.

Yeah unfortunately I view it as "exchanging hours now for days later" type of deal. I'd prefer burn through a couple of evenings this week than to be stuck dealing with this at any other point in September / October.

[IAlibay]

Ok in the interest of moving on with things, let's just go with option 1 and do a 2.6.1 release based on the current status of the repo.