ClimWIP: Failed to run preprocessor function 'mask_landsea' on the data

[DoMatte]

Hi,
I have encountered this error (see title) a few times using the ClimWIP recipe. Until now I have not find a way to fix it. I am not sure if this is for the previous written model or the the one after the line (e.g. era5).
See here:
....
2022-10-25 20:54:01,475 UTC [27144] DEBUG Running preprocessor step cmor_check_metadata
2022-10-25 20:54:01,476 UTC [27144] DEBUG Running preprocessor function 'cmor_check_metadata' on the data
<iris 'Cube' of air_temperature / (K) (time: 1032; latitude: 64; longitude: 128)>
loaded from original input file(s)
['/users/esmvaltool_tutorial/climate_data/CMIP6/ScenarioMIP/CCCma/CanESM5/ssp585/r20i1p1f1/Amon/tas/gn/v20190429/tas_Amon_CanESM5_ssp585_r20i1p1f1_gn_201501-210012.nc']
with function argument(s)
cmor_table = 'CMIP6',
mip = 'Amon',
short_name = 'tas',
frequency = 'mon',
check_level = <CheckLevels.DEFAULT: 3>
2022-10-25 20:54:01,960 UTC [27149] ERROR Failed to run preprocessor function 'mask_landsea' on the data
[<iris 'Cube' of air_temperature / (K) (time: 240; latitude: 721; longitude: 1440)>]
loaded from original input file(s)
['/users/Data/RAWOBS/Tier1/era5/tas/era5_2m_temperature_1995_monthly.nc',
'/users/Data/RAWOBS/Tier1/era5/tas/era5_2m_temperature_1996_monthly.nc',
'/users/Data/RAWOBS/Tier1/era5/tas/era5_2m_temperature_1997_monthly.nc',
'/users/Data/RAWOBS/Tier1/era5/tas/era5_2m_temperature_1998_monthly.nc']
(and 16 further file(s) not shown here; refer to the debug log for a full list)
....
Any help would be very appreciated
Thanks in advance

[remi-kazeroni]

Hi @DoMatte, thanks for reporting this issue. To help debugging this, could you please specify the full name of the recipe (if not modified compared to the public recipe) or attached the recipe to this discussion (replacing the .yml by .txt)? Could you also attach the main_log_debug.txt file that is located in the run directory of your recipe run directory?

[DoMatte]

Hi @remi-kazeroni
Thank you for taking the time and sorry for the lack of details. The recipe I am using is recipe_climwip_test_performance_sigma.yml where I modify only the set of models. Note that I also had this error with other climwip recipe. Here the debug log
main_log_debug.txt

Thanks

[DoMatte]

Dear @remi-kazeroni , @valeriupredoi
Any advice on this issue?
So, I am now understanding that the issues seem to be with the preprocessor. If I do not used the mask_landsea option, the recipe works well. But If I add in the proprocessors:
climwip_general: &general
regrid:
target_grid: 2.5x2.5
scheme: linear
mask_landsea:
mask_out: sea
then it is problematic. I have for example a
..........
RuntimeError: NetCDF: Not a valid ID
........

I am using a modify version of one of the climwip recipe using ERA5 as obs.
I attached the debug_log with this.

main_log_debug.txt

Any help will be really appreciated

[valeriupredoi]

@DoMatte could you please tell me what version of netCDF4 you have in your environment? Post the output for conda list netCDF4 please, cheers

[DoMatte]

Thanks @valeriupredoi for your quick answer.
Here it is

Name Version Build Channel

netcdf4 1.6.1 nompi_py310h55e1e36_100 conda-forg

[valeriupredoi]

indeed what I was suspecting - you must install anything but 1.6.1, 1.6.1 is known to throw these types of errors via iris. If you have built your own environment then just do mamba install -c conda-forge netCDF4=1.6.0, and if you are using a system env please tell the sys admins to not update the env dynamically and stick to the environment specs in the environment.yml file, and downgrade netCDF4 to 1.6.0 🍺

[DoMatte]

Very good! Thank you, work like a charm!
Cheers!

[valeriupredoi]

awesome, good to hear 🍺