Extracting IPCC regions

[rswamina]

I am trying to extract the newly defined IPCC-Ar6 regions from model native grid data using the extract_shape preprocessor. I used the recipe recipe_climwip_brunner2019_med.yml as an example but it is not clear to me what the id argument should like. As an example for the Mediterranean region, the string Mediterranean (MED / 19) or MED or 19 do not seem to work I donwloaded the shape file from here. What should an example preprocessor look like? I used something like this :

#recipe_extract_IPCC_AR6_regions.yml

---
documentation:
  description: |
    Extracting IPCC regions
  title: Extract IPCC-AR6 regions

  authors:
    - swaminathan_ranjini

  projects:
    - ukesm 

preprocessors:
  preproc_extract_region:
    extract_shape:
      shapefile : IPCC-AR6-shapefiles/IPCC-WGI-reference-regions-v4.shp
      decomposed : False
      method : contains
      crop: True
      ids: 
        - 'Mediterranean (MED / 19)'
    mask_landsea:
      mask_out : sea

datasets:
  - {dataset: UKESM1-0-LL, exp: piControl, start_year: 1960, end_year: 3059, grid: gn, ensemble: r1i1p1f2 }
  - {dataset: IPSL-CM6A-LR, exp: piControl, start_year: 1850, end_year: 3049, grid: gr, ensemble: r1i1p1f1 }

diagnostics:
  ext_reg:
    description: extract IPCC AR6 regions
    variables:
      tas:
        preprocessor: preproc_extract_region
        project: CMIP6
        mip: Amon
    scripts: null

    

I am using the esmvaltool module on JASMIN and the error it seems like the error means that no data points are retrieved with this id. Here is the error log :

2022-06-27 12:56:26,091 UTC [42329] ERROR   Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 796, in _run_task
    output_files = task.run()
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 263, in run
    self.output_files = self._run(input_files)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 599, in _run
    product.apply(step, self.debug)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 410, in apply
    self.cubes = preprocess(self.cubes, step,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 346, in preprocess
    result.append(_run_preproc_function(function, item, settings,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 302, in _run_preproc_function
    return function(items, **kwargs)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/_area.py", line 635, in extract_shape
    lon_min, lat_min, lon_max, lat_max = _get_bounds(
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/_area.py", line 511, in _get_bounds
    all_points = np.vstack(all_points)
  File "<__array_function__ internals>", line 180, in vstack
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/numpy/core/shape_base.py", line 282, in vstack
    return _nx.concatenate(arrs, 0)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: need at least one array to concatenate
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 499, in run
    fire.Fire(ESMValTool())
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 443, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 124, in process_recipe
    recipe.run()
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_recipe.py", line 1882, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 725, in run
    self._run_parallel(max_parallel_tasks)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 768, in _run_parallel
    _copy_results(task, running[task])
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 791, in _copy_results
    task.output_files, task.products = future.get()
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 796, in _run_task
    output_files = task.run()
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 263, in run
    self.output_files = self._run(input_files)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 599, in _run
    product.apply(step, self.debug)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 410, in apply
    self.cubes = preprocess(self.cubes, step,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 346, in preprocess
    result.append(_run_preproc_function(function, item, settings,
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 302, in _run_preproc_function
    return function(items, **kwargs)
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/_area.py", line 635, in extract_shape
    lon_min, lat_min, lon_max, lat_max = _get_bounds(
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/preprocessor/_area.py", line 511, in _get_bounds
    all_points = np.vstack(all_points)
  File "<__array_function__ internals>", line 180, in vstack
  File "/apps/jasmin/community/esmvaltool/miniconda3/envs/esmvaltool/lib/python3.10/site-packages/numpy/core/shape_base.py", line 282, in vstack
    return _nx.concatenate(arrs, 0)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: need at least one array to concatenate

[zklaus]

@bouweandela, it seems you wrote the extract_shape preprocessor initially, though those lines specifically come from @stefsmeets. Could you guys have a look or point to who might be able to help?

[rswamina]

It seems the solution is to put in Mediterranean as the id and not any of what I had tried. Currently ESMValCore only recognizes the Name property-type and not the Acronym. Here is what the dictionary entry read from the shape file looks like :

OrderedDict([('Continent', 'EUROPE-AFRICA'), ('Type', 'Land-Ocean'), ('Name', 'Mediterranean'), ('Acronym', 'MED')])

[zklaus]

Good point. Would it make sense to add Acronym to the searched fields? Perhaps open an issue about that, @rswamina.

[lukruh]

Yes, the preprocessor uses the name field of the shape file if available. I'm working with IPCC regions too and changed the line
SHAPE_ID_KEYS = ('name', 'NAME', 'Name', 'id', 'ID')
in esmvalcore/preprocessor/_area.py
to
SHAPE_ID_KEYS = ('Acronym', 'name', 'NAME', 'Name', 'id', 'ID')
to use the IPCC Acronyms for selection.

[zklaus]

@lukruh, perhaps you would like to turn that into a PR that closes @rswamina's issue 😄 ?

[rswamina]

I have created an issue ESMValCore #1648 for this now.