[Numpy2] Support for `numpy==2.0.0` #2395

valeriupredoi · 2024-04-12T15:44:27Z

Description

This PR introduces the changes we need to support Numpy 2.0 (these changes were collected from running the tests, so they may not be 100% exhaustive, but given that we have 95% test coverage, I'd say they are 95% exhaustive; we can deal with missed cases as they come along, if any; I would not use the ruff plugin test Numpy recommends to migrate, since that will introduce a lot of style changes that we may want to make when we include ruff as our linter). Most changes are in our testing suite, but there are a couple in the source code; these are backwards compatible changes, ie all tests pass when running with numpy=1.26.4

The only test that fails (both for Numpy2 and numpy legacy) is related to the pins on pandas, that need to go since we need to use the latest pandas for environment compatibility, see #2459 ; I'll try fix that issue here, since it's related to (but not a direct consequence of) Numpy2.

Closes #2460
Closes #2459

iris issue to monitor SciTools/iris#6043

The second issue closure is more of a brush it under the carpet via a pytest xfail, the original issue that kept creeping up is #2305 and the actual issue is with Pandas, and listed in pandas-dev/pandas#57002 - we can wait on its fix (and have us test quietness via the test xfail) or @schlunma can propose a more elegant solution in a separate PR?

Historical guff

OK @ESMValGroup/technical-lead-development-team we finally have a correct set of tests with the latest Numpy=2.0.0 (stable) - don't mind the ones related to pandas, I had to unpin it to gte pandas=2.2.2 built with Numpy 2.0 (otherwise the environment wouldn't solve) -> we have some work to do 👍 https://github.com/ESMValGroup/ESMValCore/actions/runs/9582595825/job/26421917358?pr=2395

~~Numpy 2.0 is impending~~: it is now fully out (see https://anaconda.org/conda-forge/numpy ); resources https://anaconda.org/conda-forge/numpy/files and https://github.com/numpy/numpy/releases - they recommend a test with the RC and list the changes roadmap in https://numpy.org/devdocs/numpy_2_0_migration_guide.html so we can test ourselves. Note: this branch should not be merged! If everything works out fine, we'll just pick up numpy 2.0 as per normal. ATTN: @ESMValGroup/technical-lead-development-team

seems the 2.0.0rc1 is a bit of a ninja - in Where is numpy=2.0.0rc1 on conda-forge? numpy/numpy#26271 it is reported that the channel is conda-forge/label/numpy_dev
moving to said conda channel results in conflicts for python-stratify, shapely, and pandas, since most prob they are pinning the upper bound of numpy to <2.0, see GA tests
a manual install is then needed; that results in mayhem - some of the fails are due to cftime, as @bouweandela points out in comments, see Numpy 2.0 incompatibility Unidata/cftime#318 (and linked PR), others are due to actual package compilation against a Numpy=1.x that barfs when it sees 2.x (those should be solved at recompile/redeploy time), and others are due to netCDF4 complaining about dtypes - that needs a closer look

codecov · 2024-04-12T15:45:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.83%. Comparing base (4b0dd41) to head (803017d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2395   +/-   ##
=======================================
  Coverage   94.83%   94.83%           
=======================================
  Files         251      251           
  Lines       14191    14195    +4     
=======================================
+ Hits        13458    13462    +4     
  Misses        733      733

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

valeriupredoi · 2024-04-12T16:23:21Z

UGH! This doesn't look good at all https://github.com/ESMValGroup/ESMValCore/actions/runs/8664961338/job/23762564320

bouweandela · 2024-04-15T08:37:18Z

xref Unidata/cftime#318

valeriupredoi · 2024-04-15T10:30:16Z

Spot on @bouweandela 🍻 Fortunately there is a mature PR for it Unidata/cftime#319

valeriupredoi · 2024-04-22T12:34:53Z

K then, just waiting for a new cftime to be released

valeriupredoi · 2024-05-27T13:11:08Z

still royally hosed even with rc2

schlunma

Thanks V, looks really good already! I have a couple of comments, mostly about removing the comments if the change is fully backwards-compatible.

schlunma · 2024-07-02T11:37:36Z

esmvalcore/preprocessor/_regrid_esmpy.py

+    # previous to numpy 2.0, a np.array(esmpy_lat.shape) was used
+    # numpy>=2.0 throws ValueError: matrix transpose with ndim<2 is undefined
+    grid = esmpy.Grid(np.vstack(esmpy_lat.shape),


Is this backwards-compatible? If yes, do we need this comment?

yes, it is, functionally and it returns the same sort of structure, only typed differently

schlunma · 2024-07-02T11:40:29Z

esmvalcore/preprocessor/_regrid_unstructured.py

+        # explicitly casting to int32 since without it, in Numpy 2.0
+        # one gets OverflowError: Python integer 360 out of bounds for int8
+        # see notes https://numpy.org/devdocs/numpy_2_0_migration_guide.html
+        src_points_with_convex_hull[-2 * n_hull:-n_hull, 1] -= np.int32(360)
+        src_points_with_convex_hull[-n_hull:, 1] += np.int32(360)


Would this work with 360.0 instead of np.int32(360)? The points should be floats, not integers, so to me this makes much more sense. If this works, you can also remove the comment

not sure, will test - but I agree about points' types indeed

nopers, a float there throws this:

> src_points_with_convex_hull[-2 * n_hull:-n_hull, 1] -= 360. E numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'subtract' output from dtype('float64') to dtype('int8') with casting rule 'same_kind'

so we either np.int32 the 360 or we np.float the left member

esmvalcore/preprocessor/_derive/co2s.py

esmvalcore/preprocessor/_time.py

tests/integration/cmor/_fixes/icon/test_icon.py

schlunma · 2024-07-02T11:47:27Z

tests/unit/preprocessor/_time/test_time.py

@@ -920,9 +920,12 @@ def test_season_not_available(self):
            name='clim_season',
            seasons=['JFMAMJ', 'JASOND'],
        )
+
+        # numpy>=2.0 these need to be explicitly cast to numpy strings
+        two_seasons = [np.str_('JASOND'), np.str_('JFMAMJ')]


I guess this is not backwards-compatible? How does the error look like? It might make sense to adapt this in the actual code so the error message still looks nice

yes it is, as it is now, ie the test does what it says on the lid for both numpy<2 and numpy>=2, here's the failed test for numpy>=2 if we don't change to forced typing:

> with pytest.raises(ValueError, match=re.escape(msg)): E AssertionError: Regex pattern did not match. E Regex: "Seasons\\ \\('DJF',\\ 'MAM',\\ 'JJA',\\ 'SON'\\)\\ do\\ not\\ match\\ prior\\ season\\ extraction\\ \\['JASOND',\\ 'JFMAMJ'\\]\\." E Input: "Seasons ('DJF', 'MAM', 'JJA', 'SON') do not match prior season extraction [np.str_('JASOND'), np.str_('JFMAMJ')]." tests/unit/preprocessor/_time/test_time.py:930: AssertionError

esmvalcore/preprocessor/_compare_with_refs.py

bouweandela · 2024-07-02T14:39:25Z

Maybe we should wait for Iris to be numpy 2 compatible: SciTools/iris#6035

bouweandela · 2024-07-03T12:35:05Z

It looks like that will happen when Iris 3.11 is released at the earliest, planned in October: SciTools/iris#5571 (comment).

valeriupredoi · 2024-07-03T12:39:47Z

many thanks for the reviews and comments @schlunma and @bouweandela 🍺 x2 I don't think we can wait until October TBF, I say we get this in and see what's the worst it can happen, and fairly sure not much, and if there are problems, we can alert iris and tell them to hurry up 😁

bouweandela · 2024-07-03T13:05:15Z

I say we get this in and see what's the worst it can happen

I'm sorry, but I have to disagree. What would most likely happen is that users will get wrong results and/or crashes.

Co-authored-by: Manuel Schlund <[email protected]>

valeriupredoi · 2024-07-03T13:16:34Z

I say we get this in and see what's the worst it can happen

I'm sorry, but I have to disagree. What would most likely happen is that users will get wrong results and/or crashes.

that is indeed the possibility; I guess there's one way to find out about the magnitude of that - run the iris tests with a numpy>=2 pin in an iris fork

valeriupredoi · 2024-09-19T14:21:30Z

Monitoring SciTools/iris#6043

valeriupredoi added 3 commits April 12, 2024 16:38

add numpy 2 from conda-forge dev

8930af0

temp remove PyPI numpy - add numpy 2 from conda-forge dev

a3e0632

run a GA

233eb67

valeriupredoi added the testing label Apr 12, 2024

valeriupredoi requested review from bouweandela and schlunma April 12, 2024 15:44

valeriupredoi added 2 commits April 12, 2024 17:13

remove numpy from env

3687ea0

manually install numpy 2.0

bf8dff5

valeriupredoi added 5 commits April 15, 2024 12:31

correct conda forge channel

29284bc

correct conda forge channel

23429bc

do not install manually numpy, like a peasant

50910b4

revert to manual installation

4d5d1eb

comment out numpy gubbins i environment yml

9fb584a

valeriupredoi closed this Apr 19, 2024

valeriupredoi reopened this Apr 19, 2024

valeriupredoi added 2 commits May 7, 2024 14:07

Merge branch 'main' into numpy2-dev

91a87ed

Merge branch 'main' into numpy2-dev

6eb3d00

valeriupredoi mentioned this pull request May 10, 2024

Pin numpy<2.0.0 to avoid pulling 2.0.0rcX #2415

Merged

4 tasks

valeriupredoi added 3 commits May 13, 2024 13:09

Merge branch 'main' into numpy2-dev

ade96ca

Merge branch 'main' into numpy2-dev

ac75b84

deactivate direct download

d36350a

valeriupredoi changed the title ~~[DNM][Test Only] Test with numpy==2.0.0rc1~~ [DNM][Test Only] Test with numpy==2.0.0rc2 May 27, 2024

valeriupredoi added 2 commits May 27, 2024 13:51

use direct download again

5dfa45a

use direct download again

baedee8

valeriupredoi added 7 commits June 21, 2024 16:33

add note on sqrt bug

a4707ef

temprary pin numpy<2

266781a

temprary pin numpy<2

73d455b

unpin and let numpy be two

3007575

unpin and let numpy be two

7386b7e

Merge branch 'main' into numpy2-dev

bf01de0

use the force and xfail the icon test instead

043407a

valeriupredoi added this to the v2.12.0 milestone Jun 24, 2024

valeriupredoi added 3 commits June 24, 2024 14:02

remove commented out pandas pins

a3ccff6

remove commented out pandas pins

aefb52c

unrun GA

1741727

schlunma requested changes Jul 2, 2024

View reviewed changes

Update esmvalcore/preprocessor/_compare_with_refs.py

4afc351

Co-authored-by: Manuel Schlund <[email protected]>

valeriupredoi added 8 commits July 3, 2024 14:21

Merge branch 'main' into numpy2-dev

6464ecf

remove numpy2 related comment

85b2aa4

remove numpy2 related comment

05dea66

Merge branch 'main' into numpy2-dev

64da79d

Merge branch 'main' into numpy2-dev

ea3c702

Merge branch 'main' into numpy2-dev

0b2e975

Merge branch 'main' into numpy2-dev

b345843

Merge branch 'main' into numpy2-dev

55edd79

valeriupredoi added 3 commits September 27, 2024 12:51

Merge branch 'main' into numpy2-dev

400ccc4

run precommit twice

e6073eb

run precommit twice

803017d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Numpy2] Support for `numpy==2.0.0` #2395

[Numpy2] Support for `numpy==2.0.0` #2395

valeriupredoi commented Apr 12, 2024 •

edited

Loading

codecov bot commented Apr 12, 2024 •

edited

Loading

valeriupredoi commented Apr 12, 2024

bouweandela commented Apr 15, 2024

valeriupredoi commented Apr 15, 2024

valeriupredoi commented Apr 22, 2024

valeriupredoi commented May 27, 2024

schlunma left a comment

schlunma Jul 2, 2024

valeriupredoi Jul 3, 2024

schlunma Jul 2, 2024

valeriupredoi Jul 3, 2024

valeriupredoi Jul 3, 2024

schlunma Jul 2, 2024

valeriupredoi Jul 3, 2024

bouweandela commented Jul 2, 2024

bouweandela commented Jul 3, 2024

valeriupredoi commented Jul 3, 2024

bouweandela commented Jul 3, 2024

valeriupredoi commented Jul 3, 2024

valeriupredoi commented Sep 19, 2024

[Numpy2] Support for numpy==2.0.0 #2395

Are you sure you want to change the base?

[Numpy2] Support for numpy==2.0.0 #2395

Conversation

valeriupredoi commented Apr 12, 2024 • edited Loading

Description

codecov bot commented Apr 12, 2024 • edited Loading

Codecov Report

valeriupredoi commented Apr 12, 2024

bouweandela commented Apr 15, 2024

valeriupredoi commented Apr 15, 2024

valeriupredoi commented Apr 22, 2024

valeriupredoi commented May 27, 2024

schlunma left a comment

Choose a reason for hiding this comment

schlunma Jul 2, 2024

Choose a reason for hiding this comment

valeriupredoi Jul 3, 2024

Choose a reason for hiding this comment

schlunma Jul 2, 2024

Choose a reason for hiding this comment

valeriupredoi Jul 3, 2024

Choose a reason for hiding this comment

valeriupredoi Jul 3, 2024

Choose a reason for hiding this comment

schlunma Jul 2, 2024

Choose a reason for hiding this comment

valeriupredoi Jul 3, 2024

Choose a reason for hiding this comment

bouweandela commented Jul 2, 2024

bouweandela commented Jul 3, 2024

valeriupredoi commented Jul 3, 2024

bouweandela commented Jul 3, 2024

valeriupredoi commented Jul 3, 2024

valeriupredoi commented Sep 19, 2024

[Numpy2] Support for `numpy==2.0.0` #2395

[Numpy2] Support for `numpy==2.0.0` #2395

valeriupredoi commented Apr 12, 2024 •

edited

Loading

codecov bot commented Apr 12, 2024 •

edited

Loading