New supercomputer at DKRZ: Levante

[remi-kazeroni]

The successor of the Mistral HPC system, Levante, is now available at DKRZ and open for usage. DKRZ plans to bring Mistral offline on May 31st. DKRZ users are encouraged to migrate to Levante as soon as possible, test their ESMValTool workflows on the new machine and copy their personal data (home directories) while Mistral is still available.

Documentation

• Documentation on Levante: https://docs.dkrz.de/doc/levante/index.html
• Current limitations: https://docs.dkrz.de/doc/levante/levante-limitations.html#dkrz-limitations-during-levante-general-availability
• Installation schedule: https://docs.dkrz.de/doc/levante/index.html#schedule-for-installation

What was done by DKRZ:

• Synchronization of CMIP data pools from Mistral to Levante
• Synchronization of project data (/work) from Mistral to Levante. User's data from Mistral are now in: /work/<project_id>/from_Mistral/<project_id>/<username>, see DKRZ info.
  Note that data are no longer synchronized and it is the user's responsibility to move newly produced data on Mistral to Levante.

What should be done by users:

• DKRZ does not handle the migration of data from /home and /scratch directories. This has to be done by the users. Note that the paths /pf/b/ will no longer exist. This is replaced by /home/b/. You may need to adjust the paths to the home directories in their script and configuration files.
• Moving data from /work/<project_id>/from_Mistral/<project_id>/<username> to /work/<project_id>/<username>.
  For both home and work data, this may be a good opportunity to clean up data spaces, free some storage space, if possible.

Noteworthy changes

• /mnt/lustre01 and /mnt/lustre02 no longer available. You can simply use: /work/ and /scratch/b/b***** on Levante
• Access to Levante is documented here. It is sufficient to substitute “mistral” by “levante” in your ssh commands. Note that there is currently no equivalent to the mistralpp nodes.
• The list and the versions of modules available on Levante differs from Mistral. In doubt, you can refer to the documentation.
• Some programs which are available by default on Mistral (emacs, git, …) can be accessed on Levante only via the ‘module load’ command (use ‘module available’ for a complete list).
• The prepost partition no longer exists on Levante. New interactive and visualize partitions are available instead. You can check their purposes and features here. Advice on data processing on Levante is available here.
• On Mistral, prepost was the only partition with Internet access. On Levante, all partitions seem to have access to Internet access and can be used to download data at runtime.

Getting started with Levante

The machine is said to be 4 times faster than Mistral with compute nodes of 256 CPUs (instead of 48) and 256 GB of RAM (instead of 64). Nodes with larger RAM (up to 1024 GB) are also available, see here.

Running ESMValTool

Together with @axel-lauer and @schlunma, we have successfully tested ESMValTool on Levante. Recipes run fine, CMIP and OBS data are available with the following configuration:

rootpath:
  CMIP6: /work/ik1017/CMIP6/data/CMIP6
  CMIP5: /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ
  CMIP3: /work/bd0854/DATA/ESMValTool2/CMIP3
  CORDEX: /work/ik1017/C3SCORDEX/data/c3s-cordex/output
  OBS: /work/bd0854/DATA/ESMValTool2/OBS
  OBS6: /work/bd0854/DATA/ESMValTool2/OBS
  obs4MIPs: /work/bd0854/DATA/ESMValTool2/OBS
  ana4mips: /work/bd0854/DATA/ESMValTool2/OBS
  native6: /work/bd0854/DATA/ESMValTool2/RAWOBS
  RAWOBS: /work/bd0854/DATA/ESMValTool2/RAWOBS
drs:
  CMIP6: DKRZ
  CMIP5: DKRZ
  CMIP3: DKRZ
  CORDEX: BADC
  obs4MIPs: default
  ana4mips: default
  OBS: default
  OBS6: default
  native6: default

PR to add support for Levante: ESMValGroup/ESMValCore#1558

[remi-kazeroni]

List of actions for our ESMValTool development workflow at DKRZ:

• Check pool of RAWOBS and OBS data
• Check that recipes work fine
• Add support for Levante in the config-file (see Add support for DKRZ-Levante ESMValCore#1558)
• Remove support for Mistral when offline (31.05.2022, done in Remove support for Mistral in config-user.yml ESMValCore#1620)
• rsync script for obs between Levante and Jasmin (@remi-kazeroni: working fine)
• Setup a common directory for automatically downloaded data (restricted to ESMValTool development project members)
• Check that ESMValBot still runs fine
• Any other service / project folder that is Mistral related?

Feel free to expand the list if needed.

[remi-kazeroni]

DKRZ has confirmed that Mistral will be decommissioned end of the month, see announcement. Please all switch to Levante asap and comment in this discussion if you encounter issues running ESMValTool on that machine.

[remi-kazeroni]

Also currently missing on Levante:

• ESMValTool module
• ESMValTool Jupyter kernel

I'm not sure DKRZ staff would have time to work on that now (see discussion in #1888). Let's try to have those back for the next release at the latest.

[zklaus]

That's Jupyter kernel, right?