Should NetCDF files created by ESMValTool follow the CF Conventions? #2643
Replies: 11 comments 21 replies
-
|
Yes, they should, definitely. Do you have some examples that don't and in which way they deviate? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, but then we have to change all effected diagnostics to use the long name instead of the standard name. And eventually follow up diagnostics (through ancestors), to make them still pick the right data. If there are many non CF compliant netCDF files at the moment, that could require changes in many diagnostics. |
Beta Was this translation helpful? Give feedback.
-
|
Could you give an example for a file/diagnostic where this is happening? |
Beta Was this translation helpful? Give feedback.
-
|
How do you check if they files are CF compliant? I found that online checker: https://pumatest.nerc.ac.uk/cgi-bin/cf-checker.pl |
Beta Was this translation helpful? Give feedback.
-
|
According to the CF website, there are also two Python-based checkers, see http://cfconventions.org/software.html |
Beta Was this translation helpful? Give feedback.
-
|
I just provided examples in #2643 (comment). |
Beta Was this translation helpful? Give feedback.
-
|
I agree with @katjaweigel . It has never been our policy that output from diagnostics has to be CF compliant. The idea was to simply save the "numbers" used to create a plot. |
Beta Was this translation helpful? Give feedback.
-
|
As I explained above, @katjaweigel's comment is orthogonal to the question at hand. We certainly claim for many files that they are CF compliant since we include the |
Beta Was this translation helpful? Give feedback.
-
|
Having the units and variable names along with the "numbers" would help with understanding the output though. This would also be useful if other tools want to build on results computed by ESMValTool (e.g. further figure editing in a Jupyter notebook, or displaying results interactively on a webpage if we ever get to that). |
Beta Was this translation helpful? Give feedback.
-
|
Actually thinking about it, this might need a bit of differentiation? I don't see the need for CF compliancy in files that are further used internally (like files saved by one diagnostic when using the ancestory dynamic) or files that are only auxiliary files that are only documenting the values of plots. However diagnostics that output files specifically made to use later on with external tools etc I could imagine CF compliancy being useful. So what is the current goal of netcdf files produced? They might be wildly different. |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps. At least with Iris, but probably with other support libraries as well, CF compliance should be basically automatic, so I think we really need to have some examples of how files are not following the conventions to understand whether that is reasonable, something that is easily fixed, or perhaps something that points to a problem in the conventions themselves. |
Beta Was this translation helpful? Give feedback.
-
|
I just provided examples in #2643 (comment). These examples are taken from the |
Beta Was this translation helpful? Give feedback.
-
|
Well, the documentation for the Core says "The CF Conventions as well as additional standards imposed by CMIP should be followed whenever possible." For diagnostic contributions to the tool there is no such recommendation (not even speaking about a rule). I agree with @bettina-gier, I also do not see the point of such a requirement for diagnostics. We could think about encouraging people but no need for making this a requirement. |
Beta Was this translation helpful? Give feedback.
-
|
iris accepts and outputs only CF-compliant data, so I'd be curious too where the issue is (as in, what actual data is not CF-compliant and still goes through our pipeline and gets outputted in non-CF-compliant format) |
Beta Was this translation helpful? Give feedback.
-
Touchy if you asked me, that means that data would not be usable with iris (but maybe with cf-python, definitely with xarray) so one needs to think a bit before |
Beta Was this translation helpful? Give feedback.
-
|
Hmm... this sounds like the whole world is using exclusively Python and Iris... ;-) The main purpose of the netCDF output from the diagnostics was documentation to help with reproducibility, so one could, for example, check the exact numbers shown in a plot when trying to reproduce a figure. In most cases, it was never intended that someone else could easily read the output with a specific software package (e.g. Python/Iris). Unless this should change, I do no see a good reason to put an additional constraint on diagnostic developers. |
Beta Was this translation helpful? Give feedback.
-
|
Quite the opposite: Because the output should be easily readable with any of the many software tools, it should follow the applicable standard and not be guided by any particular software or any one person's intuition. Of course, if the software helps adhere to the standard all the better. I personally read CF compliant Netcdf files with Python and Iris, but also frequently with Ncview, sometimes with Qgis, Paraview, and Blender. I would like to get more familiar with Panoply and Psy-view. So I agree with you: The goal is not to target a specific software package. That's why it should be standards compliant. |
Beta Was this translation helpful? Give feedback.
-
|
absolutely! If we are guided by the "file is there for posterity, format not important" principle, we are running quickly into what the Egyptians never thought of - that it would take until the 19th century for someone to decode their silly picture language 😆 |
Beta Was this translation helpful? Give feedback.
-
|
also - I consider iris ahead of its time wrt this - standardization is very good to have, even if that introduces restrictions, it pays off in the long term |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, sorry to barge in un-invited, FYI, the IPCC task group on data support and climate change assessments (TG-Data) recommends IPCC authors use community standards for data formats and metadata attributes when creating IPCC content. If the diagnostics that are discussed in this issue are likely to be displayed in an IPCC report graphic, this would strenghten the case for CF-compliance. Thanks again for your work on ESMValTool, |
Beta Was this translation helpful? Give feedback.
-
|
cheers @huard - excellent point! And we have quite a few diags that produced plots for AR6 so yes for CF-compliance methinks! |
Beta Was this translation helpful? Give feedback.
-
I investigated this in a bit more detail and wrote a small script to analyze the situation. For v2.5 this led to the following results:
Here is the code of the script, in case anyone wants to dive into this in more detail. import sys
from pathlib import Path
import iris
from cfchecker.cfchecks import CFChecker, CFVersion
def find_files(dirname):
return Path(dirname, 'work').glob('**/*.nc')
def _check_with_iris(filename):
try:
iris.load(filename)
except Exception:
return False
return True
def check_with_iris(dirname):
return all(_check_with_iris(f) for f in find_files(dirname))
def check_with_cfcheck(dirname):
checker = CFChecker(version=CFVersion(), silent=True)
for filename in find_files(dirname):
try:
checker.checker(str(filename))
except Exception:
print("Unable to check file.")
return False
totals = checker.get_total_counts()
err_levels = (
'FATAL',
'ERROR',
'WARN',
)
errs = sum(totals[k] for k in err_levels)
if errs:
# Only print info for files that actually have a problem
for filename, file_result in checker.all_results.items():
result = file_result['global']
msg = []
for level in err_levels:
msg.extend(f' - {level}: {problem}'
for problem in result[level])
if msg:
msg.insert(0, ' global netCDF attributes:')
for var, result in file_result['variables'].items():
var_msg = []
for level in err_levels:
var_msg.extend(f' - {level}: {problem}'
for problem in result[level])
if var_msg:
var_msg.insert(0, f" netCDF variable '{var}':")
msg.extend(var_msg)
if msg:
msg.insert(0, f' {filename}')
print('\n'.join(msg), flush=True)
return not errs
def check_results(dirnames, check):
success = 0
fail = 0
for dirname in dirnames:
print(f"Checking {dirname}:", flush=True)
if check(dirname):
print(f"{dirname}: Success", flush=True)
success += 1
else:
print(f"{dirname}: Fail", flush=True)
fail += 1
print(f"Success: {success}, Fail: {fail}")
if __name__ == '__main__':
if sys.argv[1] == 'iris':
check_func = check_with_iris
recipe_dirs = sys.argv[2:]
else:
check_func = check_with_cfcheck
recipe_dirs = sys.argv[1:]
check_results(recipe_dirs, check_func) |
Beta Was this translation helpful? Give feedback.
-
|
Here is yet another summary. The first column lists the number of affected recipes and the second column the problem as reported by cf-checker:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the summary!
and I know that all diagnostics in there are using python and IRIS to save the data. The cell method is for example the name of an IRIS Aggregator I wrote. So far I was not aware:
My other recipes with python diagnostics seem to pass here, e.g.:
although with the online checker I mentioned above I also got a warning on a certain files type from this recipe:
|
Beta Was this translation helpful? Give feedback.
-
That's interesting. BTW if we would like Iris to be more helpful in this regard, we may be able to get this on the team's request list under IS-ENES3 funding. So worth thinking about Iris requirements requests in this context. |
Beta Was this translation helpful? Give feedback.
-
|
Looking at the list (thanks @bouweandela for this awesome summary!), I think we can categorize them loosely as follows: Missing
|
Beta Was this translation helpful? Give feedback.
-
|
if we want to have us a discussion about cf-checker codes, it's prob best to ping @RosalynHatcher who is the developer of the cf-checker used a lot in various places - it wouldn't hurst to have that checker run part of our workflow too, should the user desire 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Great to see this discussion. With my techy-scientist hat on, I think that file conventions are important and in general the effort of setting sensible metadata outweighs the pain of trying to guess what is in a file. So personally I'd like to see adherence to CF conventions in output & intermediate files encouraged in the development workflow, provided that there are tools & guidance to make it easy to get this right (e.g. interfaces that make it easier to write compliant files for all of the primary languages we support). With my "backward incompatibility discussion" hat on, this would make a good case study as it raised lots of interesting questions. E.g. if we did decide that this was desirable, then:
#everyoneloveshats 🎩 👒 🎩 |
Beta Was this translation helpful? Give feedback.
-
While developing a tool for comparing the results of running an ESMValTool recipe across different versions of the tool (#2613), I noticed that many diagnostics produce NetCDF files that do not comply with the CF Conventions. Does anyone have an opinion on this? Should the NetCDF files produced by diagnostics follow the CF Conventions, or is it not important? @ESMValGroup/esmvaltool-developmentteam
Beta Was this translation helpful? Give feedback.
All reactions