Re-using pre-processed files or other previous run output?

[senesis]

When tuning a script, or when studying the sensitivity of a result to some script parameter, it could be handy to avoid repeating the pre-processing of input data. Is there a way ?

[zklaus]

Yes! In fact, there are two ways: If the pre-processing has finished completely, you can just run the diagnostic script. For this, ESMValTool will give you the exact command line to use in the logs towards the end (look for "to re-run the diagnostic..." or similar.

If the pre-processing hasn't finished completely, you can resume an earlier run. This is a feature that was added in 2.4.0 (see the changelog) and is described in the documentation.

[senesis]

Thanks. That's useful. However, the user has to explicitly handle the re-used output directory names, which may be uneasy.

A remark : next doc section is not clear about how to specify the machine for a directory :

Another potential use case is running the preprocessing part of a recipe on one or more machines that have access to a lot of data and then running the diagnostics on a machine without access to data.

[senesis]

The 'resume' feature could be useful also for reusing ancestor tasks output when such tasks succeeded. Is it already handled, or would it require more development ?

[bouweandela]

Yes, this is already implemented, but only for preprocessing tasks.

[seyedmohammad19]

Hi I have a question:
I downloaded data from ESGF. before running in ESMVALTOOL, does the data need to be preprocessed? like downscaling and bias correction?

[bouweandela]

There is no need to preprocess the data before running the esmvaltool command, ESMValCore will take care of that (this can be specified defined in the recipe). For an overview of how the ESMValTool works, I would recommend having a look at this introduction.

[bouweandela]

Also note that the esmvaltool command can automatically download any data required (provided it is on ESGF) to run a recipe by using the --offline=False command line option or specifying offline: false in the user configuration file.

[seyedmohammad19]

Thank you very much