Should netCDF support GRIB2 as a read-only dispatch layer (as it supports HDF4)?

[edwardhartnett]

GRIB2 is the WMO format for weather data. It's a complex and challenging standard. One key characteristic is it's dependency on code tables to interpret data. There are some WMO global tables, but also local tables that anyone can add. This informality means that the same code can mean different things in different files. Some documentation can be found here: https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/.

NOAA has several repositories of GRIB2 tools and libraries, in C and Fortran. These are somewhat basic and are not meeting the current needs of NOAA scientists in various ways. They need a serious update to clean up and add missing features. These libraries are part of the Unified Forecast System (UFS) and other important operational products, including the NOAA Aviation weather forecasts.

I am now updating all these NOAA GRIB2 libraries and tools, and am adding the lacking features to make them more useful to NOAA moving forward. Most of this work is taking place in the NCEPLIBS-g2c library.

When I'm done, this will be easy to package up as a netCDF dispatch layer. Most of NOAA's GRIB2 needs have to do with converting between netCDF and GRIB2. So having the netCDF library understand GRIB2 would have significant benefit. Conversions from GRIB2 to netCDF would become unnecessary, and all netCDF-based tools and code would work with GRIB2 files. All netCDF users would instantly have access to many new meteorological data sets, with no extra work at all!

Wouldn't that be cool?

I will get this working as an independent project, using the user-defined format feature. But should this be part of netcdf-c? I rather think it should.

One concern is going to be to present the data the same way that netcdf-java would present it. If there's anyone at Unidata familiar with how netcdf-java interprets GRIB2 files, I would very much welcome some guidance.

@DennisHeimbigner and @WardF what are your thoughts on this.

[DennisHeimbigner]

Ed-
I have contacted Hailey and Tara, our netcdf-java people, to chime in.
Is there some reason why using thredds to manage the grib
and then using netcdf-c to read that as, say, dap4, is not acceptable?
I would hate to have to redo the rather complex netcdf-java grib code.

[edwardhartnett]

Using netcdf-java is not an option for most large data producers, like NOAA. Everything is done on HPC systems, and Java is never installed or approved.

Currently most NOAA users use a handful of (FORTRAN) utilities to convert GRIB2 to netCDF or vice-versa.

[dopplershift]

cc @haileyajohnson @tdrwenski

I don't have any strong opinions on whether, once it exists, it should be included. I will ask, though, if it can be done completely independently as a read-only plugin, what the value proposition is for having it included rather than remain its own separate, useful thing? This would mean netCDF-c depends on the NOAA/ECMWF GRIB libraries or picking up (yet another) optional dependency. I know those have been making things complicated (in terms of testing combinations) lately. 🤷‍♂️

I want to first draw attention to the other option in this space, the Python-based cfgrib library from ECMWF. I'll note that it and netcdf-java took different approaches to mapping GRIB to netCDF.

The challenges I'm aware of in doing this mapping:

1. There's no such thing as a GRIB2 file. GRIB2 is a format for messages holding horizontal 2D slices of data, and files can be composed of arbitrary sets of these messages. This leads to challenges to coming up with shared dimensions and handling the cases where certain fields are missing messages for certain times/levels.
2. Field names. GRIB2 parameters only describe the field (e.g. "air temperature"). A full model run can have air temperature on a bunch of different vertical coordinates (2m, isobaric levels, certain sigma levels). Converting those to unique variable names within the dataset is tricky. netCDF-java chose to make gigantic, unique names. cfgrib chose to only allow you to select one.
3. Fields like precipitation have the same "coordinate" issue as above, but for time. The same parameter id is given to 3-hour, 6-hour, etc. totals. Turning those into a sensible and useful netcdf variable is tricky.

Don't take those as dissuasion, just passing along the pain points I've watched others struggle with.

[edwardhartnett]

Don't worry, you won't dissuade me. ;-)

To respond to your issues:
1 - A GRIB2 file contains one or more GRIB2 messages (and maybe other stuff that is ignored). Grid definitions are only valid within messages - there are no file-global dimensions (and the same is true in HDF4 and HDF5). I intend to handle it in the same way I handle HDF5 and HDF4 files: if a file contains a bunch of messages with identical grid definitions, I will assume that is a set of shared dimensions.
2 - NOAA has specified field names (long and short) for all the parameters covered in the WMO global tables, and NOAA's local tables. This is all built into the g2c library and has been (and will be) kept up to date with any changes or additions.
3 - Time is indeed tricky. In my first pass I will expose the GRIB2 time information as attributes, in a future pass it may be possible to convert it to match CF conventions.

Due to the User-Defined Format feature, this does not need to go into netcdf-c for NOAA. I am offering it because I believe it will benefit other netcdf users, just as the HDF4 dispatch layer has.

[dopplershift]

I think you're over-simplifying the use of grid definitions and missing the part where there are missing messages, but maybe that's not a problem when on NOAA HPC.

I think you'll have a lot of success with the reduced problem space that is being NOAA-HPC-specific.

I think solving the problem more generally, suitable for inclusion in NetCDF-c means dealing with ALL GRIB (MRMS, ECMWF, etc.), will end up with all the same challenges we've seen and continue to see in netcdf-Java land.

[edwardhartnett]

The addition of ECMWF and MRMS is straightforward. If there are others, please list them.

Missing values are missing values. Does netcdf-java do something different?

Some see possibilities and do, others see difficulties and don't. Both are valid viewpoints, but it's the former that led to all the great features netCDF now contains.

[dopplershift]

Ed, I'm here really just trying to point out the hard-won lessons I've seen smart people here learn with a great deal of pain. Not trying to be difficult, not trying to dissuade, just trying to helpfully point out the pitfalls and help you avoid some of that pain. I truly believe the problem is harder than you think. It's also possible I'm wrong, which would be great news, because we'd have an even better approach for the mapping and the community would win. Just knowing who's worked on it before, I just find it unlikely.

Some see possibilities and do, others see difficulties and don't. Both are valid viewpoints, but it's the former that led to all the great features netCDF now contains.

But seeing as how you want to sermonize here and talk down to me, I'm out. Have fun and good luck!

[edwardhartnett]

I have been sermonizing on netCDF since you were an undergrad, and am unlikely to stop any time soon. ;-)

It's easy to list problems - it's more difficult, but more helpful, is to suggest solutions. However, if you feel more comfortable not participating in this effort, that's fine.

Certainly there is much work to do, and I welcome your contribution to netCDF, in whatever area you choose.

[DennisHeimbigner]

Ed- is there any kind of written specification for the api provided by the current or proposed NOAA grib libraries?

[edwardhartnett]

There is full doxygen documentation here: https://noaa-emc.github.io/NCEPLIBS-g2c/.

The built documentation is for the current release of g2c, which does not yet include the new functions. But they are fully documented as well and, when released, that documentation will be on-line.

Currently the g2c V2 API is under development and not yet stable. Give me until the end of the year for that...

It should be noted that the g2c library has been around for a very long time - it's certainly older than netCDF. As part of the operational systems it is maintained by a team here at NOAA and that will continue as long as it is used in operational or research systems, so for many years and probably decades to come.

[dopplershift]

I guess one other thought comes to mind: if this were to land in NetCDF-c, it should probably have identical behavior to NetCDF-Java to avoid confusing our users.

[DennisHeimbigner]

it should probably have identical behavior to NetCDF-Java to avoid confusing our users.

This will probably not be entirely possible since CDM is not quite the same data model
as netcdf-4. But since (I assume) grib for netcdf-java probably does not use structs or other user-defined types, then the differences should be small.

[edwardhartnett]

I believe it will be possible. I intend to interpret the file, as netcdf-java does, as a netcdf-classic file (if I understand netcdf-java correctly).

Where are the netcdf-java GRIB2 test files? Is there any easy way for me to get the CDM version of their metadata?

[czender]

Grib2 is here to stay for decades (much like HDF4). It sounds to me like there is strong institutional (NOAA) support for the initial implementation and perhaps long-term maintenance. If the implementation is up to netCDF standards (and given Ed's involvement that seems almost guaranteed) then I think this would be a win for users. However, there must be many competing demands on the time of the core team, so Grib2 would also have to make sense in terms of the opportunity cost of the core team's time, which I have little insight about.

[DennisHeimbigner]

There are, I assume two parts to the grib support.

1. grib processing libraries
2. the netcdf dispatch code for grib data.

I assume No. 1 is NOAA's responsibility
So the question is how hard is it to build and maintain no. 2

[edwardhartnett]

The NOAA GRIB libraries and utilities have long-term institutional support from NOAA. They contain some code that is decades old. They are supported as part of the NCEPLIBS libraries, at NOAA EMC. There are about 5 of us on the team, but NCEPLIBS includes many projects besides GRIB libraries. ;-)

NOAA now uses netCDF for internal data processing in the UFS (hooray!), but is still required to translate a lot of the final output data into GRIB2. So this library is not going anywhere.

GRIB2 is a WMO standard, and it's built into a LOT of operational software and hardware systems. As Charlie notes, it will be around for a decade at least. (Much longer, I suspect.) Despite John's negative views of GRIB2 (some of which I certainly share), it is much beloved in the atmospheric science community, mostly because it has really excellent compression. (The recent addition of quantize and zstandard to netCDF certainly steals some of this thunder, but there is more that can be done.)

@DennisHeimbigner my idea is to write the dispatch layer code as part of the NCEPLIBS-g2c library. Once it is working well, then we could move it to netcdf-c. Since I am also writing the GRIB2 library, you can be sure that I will include the functionality I need to make the dispatch layer very simple.

As @dopplershift notes, the real challenge will be getting similar or identical answers as netCDF-Java. That's a more long-term goal.

[DennisHeimbigner]

Personally, I think you both make good arguments for adding netcdf support for grib.
The problem is, as always, is finding the resources for it. Assuming Ed built the relevant
dispatch code, it will probably fall to Unidata to support that dispatch code.
Is there any chance that we could find some funding for this somewhere?

[edwardhartnett]

The help I am giving is the NOAA support. ;-) I do not have any funding to provide.

However, I don't think maintaining the GRIB2 dispatch layer will be too demanding. All the functionality, including translation of GRIB2 codes into (hopefully CF compliant) attributes will live in the NCEPLIBS-g2c library. That functionality will be used in various NOAA utilities, so it will not live in netcdf-c.

Of course none of us can know the future level of support for any of this. Unidata may be sold to PizzaHut, the NCEPLIBS team might all be sent to fight climate change-induced flooding of Florida, and NOAA might be replaced by an AI run by Amazon on a cloud of quantum computing servers on Luna. In which case you can come up to my place and help me grow cabbages and make beer.

I suspect I will not really be ready with any serious netCDF compatibility capability at least until mid-2023. At that time you can see what I have and we can discuss based on more detail.

[WardF]

@DennisHeimbigner there are a handful of RFP's out now that we might be able to write something for; I'll need to take a closer look to make certain, once I'm done preparing my AGU materials :).

[HathewayWill]

I feel that anything that will help make programs and software more usable is a good thing. The atmospheric community is riddled with different file formats and encoders so expanding NETCDF to include more of these will help users around the globe.

It is important to remember that the scientific community is global and not to just focus on Europe and North America.

[fcvdb]

What is described:

So having the netCDF library understand GRIB2 would have significant benefit. Conversions from GRIB2 to netCDF would become unnecessary, and all netCDF-based tools and code would work with GRIB2 files. All netCDF users would instantly have access to many new meteorological data sets, with no extra work at all!

Would be a great feature.

The NCAR NCL executable ncl_convert2nc is a great function to read GRIB2 data, but the ability to write in GRIB2 format has always been lacking.

[edwardhartnett]

An update: there has been a lot of progress in this area.

One area of progress has been our teaming up with @webisu, author of the wgrib2 package. wgrib2 is extremely capable and deals with many codes. It has an advanced API which seems very capable of supporting the netCDF-GRIB2 dispatch layer. Wgrib2 is NOAA operational software which manipulates GRIB2 files and messages. It's used at NOAA to translate GRIB2 files to netCDF.

However, I was overoptimistic as to schedule. We now plan to have our new g2c API out by the end of this year. Between g2c and wgrib2, we will have all the functionality needed for a netCDF integration layer, which I can release as a User-Defined-Format package. Then we can discuss possible inclusion in netcdf-c.

[czender]

That is fantastic news, Ed. Thanks for persevering with this project @edwardhartnett and @webisu.