New ESMValTool installation does not reproduce output from standard recipes

[TomasTorsvik]

Hi,

I am trying to install ESMValTool, and follow the guidelines for installation with mamba as described in the documentation https://docs.esmvaltool.org/en/latest/quickstart/installation.html#mamba-conda-installation

The installation goes through without problems, and I get a working esmvaltool application

$ esmvaltool version
ESMValCore: 2.8.0
ESMValTool: 2.8.0

However, the output I get when running standard recipes are not the same as the reference output. For instace, the recipe_ocean_quadmap.yml example:
Reference: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.0/recipe_ocean_quadmap_20230321_120309/
My installation: http://ns2980k.web.sigma2.no/tomast/esmvaltool2.8.0/recipe_ocean_quadmap_20230512_132526/index.html

I have now tried the installation on two separate systems, and I get the same result on both. The recipe still works with an old installation of esmvaltool v2.7.0, but fails in the same way when I use a new esmvaltool v2.7.0 environment.

I have also tried to install from source, and set iris to version 3.4.1, but that did not solve the problem. I hope someone here in this form may have a suggestion about what can be wrong, and suggestions for what I can try to fix it.

[ehogan]

Hi @TomasTorsvik! 👋 Thank you for getting in touch! 😊

I ran the recipe_ocean_quadmap at my site (Met Office) and the output I get looks the same as the reference you linked to in your comment. I created the ESMValTool v2.8.0 environment in April:

2023-05-17 06:54:09,534 UTC [69159] INFO    Package versions
2023-05-17 06:54:09,534 UTC [69159] INFO    ----------------
2023-05-17 06:54:09,534 UTC [69159] INFO    ESMValCore: 2.8.0
2023-05-17 06:54:09,534 UTC [69159] INFO    ESMValTool: 2.8.0
2023-05-17 06:54:09,534 UTC [69159] INFO    ----------------

The environment on JASMIN also produces an output that looks the same as the reference (@valeriupredoi updated the version of ESMValTool on JASMIN recently, see #3163, but there was already an environment available, so I'm not sure whether any dependencies were updated). [Edit: I just noticed @valeriupredoi said he updated the version of Iris to v3.5, see this comment on #3163, so some dependencies will have been updated.]

I also note (from the logs):

• Reference:

2023-03-21 12:03:09,777 UTC [2225457] INFO    esmvalcore._main:496 Package versions
2023-03-21 12:03:09,777 UTC [2225457] INFO    esmvalcore._main:497 ----------------
2023-03-21 12:03:09,777 UTC [2225457] INFO    esmvalcore._main:498 ESMValCore: 2.8.0rc2
2023-03-21 12:03:09,777 UTC [2225457] INFO    esmvalcore._main:500 ESMValTool: 2.8.0.dev111+g6faf263f6
2023-03-21 12:03:09,777 UTC [2225457] INFO    esmvalcore._main:501 ----------------

• Your installation:

2023-05-12 13:25:26,597 UTC [3519507] INFO    esmvalcore._main:496 Package versions
2023-05-12 13:25:26,597 UTC [3519507] INFO    esmvalcore._main:497 ----------------
2023-05-12 13:25:26,597 UTC [3519507] INFO    esmvalcore._main:498 ESMValCore: 2.8.0
2023-05-12 13:25:26,597 UTC [3519507] INFO    esmvalcore._main:500 ESMValTool: 2.9.0.dev0+g9dcaa5575.d20230512
2023-05-12 13:25:26,597 UTC [3519507] INFO    esmvalcore._main:501 ----------------

This means the problem is either related to the fact that a different (perhaps recently released) version of a dependency is selected and installed in the environment you just created, or that the version of ESMValTool (2.9.0.dev0+g9dcaa5575.d20230512) contains changes (compared with 2.8.0) that are causing problems (although I'm not sure what those would be?).

Does your environment compare to the environment specified by the conda-linux64.lock file? I believe @remi-kazeroni was planning to run all recipes using an environment that @valeriupredoi will create over the next few days, so we could see whether the problem occurs there too? If there are problems, that means there will be problems for all users creating environments, but if not, then that means there is something specific going wrong with your installation (which hopefully we can figure out!).

A potential workaround in the meantime could involve creating a lock file from e.g. the JASMIN installation, and using that to create an environment?

I hope that helps? 😊

[TomasTorsvik]

Hi @ehogan , thanks for quick response!

It seems that the example I sent was from the experiment I did when building from source, but I get the same output for the 2.8.0 version.

Thanks for pointing me to the lock file! I was able to get a working installation sing the instructions from
https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/install.html#installation-from-the-conda-lock-file

Now I get the same output as before
http://ns2980k.web.sigma2.no/tomast/esmvaltool2.8.0/recipe_ocean_quadmap_20230517_085108/index.html
I think this can be a workaround I can use for now.

I get a lot of missmatches and some missing packages when I compare my original environment with the conda-linux64.lock file. Probably something here could be significant, but I'm not sure what to look for
env_compare.txt

[valeriupredoi]

Hi @TomasTorsvik and @ehogan - am currently at a conference until Friday, when I will have to release a release candidate for Core, for a bugfix release, but I will definitely have a close look at this next week. In the meantime could you pls produce the env record of the environment that produces those funky results, and attach it here? You can do that with conda env export > funky-env.yml Cheers 🍺

[TomasTorsvik]

Hi @valeriupredoi
here is the funky environment that I got when using the mamba installation procedure
funky_env.txt
Cheers

[valeriupredoi]

Awesome, cheers, mate! the only thing that I see that might be the smoking gun is the new iris=3.5 (matplotlib's been at 3.7.1 since 2 months now) - would you be able find a bit of time and just downgrade iris with conda install -c conda-forge iris=3.4.1 and rerun, please, mate? Cheers 🍺

[TomasTorsvik]

Hi @valeriupredoi ,
I created an environment with iris=3.4.1, but it's still funky
env_funky2.txt

See output here
http://ns2980k.web.sigma2.no/tomast/esmvaltool2.8.0/recipe_ocean_quadmap_20230517_164056/index.html

[valeriupredoi]

Agh! Many thanks @TomasTorsvik - very useful, I'll have to give it a closer look then, tomorrow. Cheers for all your investigations 🍺

[valeriupredoi]

Probably best to escalate this to an Issue, will do this tomorrow 👍

[remi-kazeroni]

I was able to reproduce the same funky plot by using the module v2.8.0 on Levante. This module uses the base code from the last release (that was used to test all recipes) but was deployed early May (so with more recent versions of some dependencies). Taking a closer look at preprocessed files, I see that the bugger is preproc/diag_map_1/tos/obs4MIPs_ATSR_tos_2001-2003.nc (CMIP5 files look ok) which contains lots of e+19 values, hence the funky plot.

It seems that the problematic dependency is numpy=1.24.3, not iris. Making a fresh new environment, one gets numpy=1.24.3 (see here for conda install). Downgrading numpy with mamba install numpy=1.24.2 (no other dependency changed) was enough to get the "normal" plot. Not sure exactly what's wrong in this case...

@valeriupredoi - shall we think of pinning numpy to numpy !=1.24.3 as this was done for the Core?

Does your environment compare to the environment specified by the conda-linux64.lock file? I believe @remi-kazeroni was planning to run all recipes using an environment that @valeriupredoi will create over the next few days, so we could see whether the problem occurs there too? If there are problems, that means there will be problems for all users creating environments, but if not, then that means there is something specific going wrong with your installation (which hopefully we can figure out!).

@ehogan - Yes, but I was only planning on running the recipes and count the number of "Run was successful" in the log files. I was not really planning on checking the output of the recipes manually, I don't really have the time for that. But this problem here shows that we should be careful about recipe output

[valeriupredoi]

@remi-kazeroni very good detective work! That numpy vrrsion is known to have a severe bug in masking, we pinned it in Core, but never got around to do the same in Tool, hence I didn't even look at numpy in Tomas' env. Yes, we should defo pin Numpy !=1.24.3. Bugger, this thing was bothering me, well done 🍺

[remi-kazeroni]

Was bothering me as well. It's probably affecting more recipes but I do not have the capacity to run many tests and check the output manually... Ok, I can open a PR to pin Numpy in the Tool env. But what about conda-forge? I think we should pin there as well. Otherwise users will keep getting 1.24.3 like @TomasTorsvik.

[valeriupredoi]

OK finally in front of my computer at work - I finally figured this kerfuffle out - excellent input from @TomasTorsvik @ehogan and @remi-kazeroni - the installation from conda is still picking up iris=3.5 or below (currently 3.5 see install log ) and they have only introduced the pin on broken numpy=1.24.3 only yesterday in iris=3.6; we, on the other hand, have the pin in Core but only in the dev main branch, so Tomas faced a perfect storm: the conda install is rather too up to date wrt numpy, not so up to date wrt iris (which is normal), and we didn't release the new Core with our pin in there, so he (and our conda install tests) pick up the broken numpy; what's more alarming is that I passed by the numpy conda feedstock, and there is not even a trace of a numpy=1.24.4 with a bugfix, so we should definitely pin it in Tool, even if the pin will come from iris from today onwards, and from (a possible new) Core in the near future - just to be safe 🍺

[remi-kazeroni]

Thanks a lot for taking the time to report and investigate this issue @TomasTorsvik! This was likely to affect many of our users. The issue was fixed by @valeriupredoi in #3182 (thanks very much V!).

@TomasTorsvik, all should now work fine with a fresh new environment. Feel free to reopen this discussion if needed!