Skip to content

Notes on History TNG

Jump to bottom
William Jamieson edited this page Mar 11, 2021 · 8 revisions

Notes on History, The Next Generation

These are @WilliamJamieson's notes on History TNG to help document the current progress on this effort.

Overview

History TNG is intended to be a replacement for the current GEOS history component, which utilizes NUOPC's "asynchronous" capabilities. In theory, NUOPC should (or will be extended to) allow us to perform similar IO functions as our IO-server without the need for our current IO-server. Moreover, this process affords us the opportunity to "reimagine" how GEOS history operates and works.

Currently, GEOS history has accumulated lots of different use cases/configurations some of which have been added in a very ad hoc fashion over its lifespan. Supporting all of this while allowing for the ability to add new features is becoming increasingly difficult. Especially, due to the fact that choices in the configuration format are starting to become self-limiting. Hence, part of creating History TNG includes a thorough survey of the current GEOS history features, so that History TNG can be built so that it can accommodate all of these features in a sustainable way. In particular, this includes a redesign of the History input file from the HISTORY.rc file into a History.yaml file including the creation of a flexible infrastructure to parse the new file format in an easily extendible fashion.

I see three central goals of the History TNG project:

  1. Creating an NUOPC based history component (or set of components).
  2. Migration of the IO-server's capabilities into NUOPC.
  3. Creation of an easily extendable yaml import format together with solid documentation of its limitations.

Note that I consider a functional history component to be a configurable component which can take in a collection (or set of collections) and write all the fields in that collection to a file.

Roadmap

Rocky Dunlap (@rsdunlapiv) has proposed what he calls the "scooter", "car", etc. method of developing History TNG. By this, he means that after every major developmental step (and hopefully at every point in between) we maintain an operational history component (not full featured, but a functional history component).

So the steps in this roadmap are as follows:

  1. Create a "scooter" which is a history component which takes in an input file containing a single collection, and then writes that collection to a file. There should be no fancy features, basically writing all the fields in the collection to a file with no re-gridding, interpolation, or other "fancy" operations. This will be a severely limited history component, with the basic goal of creating a clean NUOPC based History Writer component together with the necessary changes to GEOS to allow NUOPC to facility the coupling of the components.
  2. Create "scooter 2.0" which is the same as "scooter" accept that the model allows one to add multiple collections, via the creation of additional History writer components.
  3. Create "Motorcycle". This would introduce the basic re-gridding and interpolation functionality that history currently supports. This could be done by adding NUOPC mediator components between the History Writer Components and GEOS. NUOPC mediators are intended to allow NUOPC to implement automatic re-gridding, interpolation, and other model interoperability features.
  4. Create "Car". Here one might introduce a "main" history component which exists on both the history GEOS pet lists. This main history would attempt to orchestrate how all the History Writers are operating by assigning collections to different writers as needed. Here is where we start introducing IO-server capabilities into History TNG.

Note that this is a rough road map, which should be refined over time. In particular, all of the features that history currently supports should be implemented when appropriate in this timeline (I have a feeling that "motorcycle" will go through several iterations). The big question for me is if the NUOPC connectors are flexible enough to support the features that "car" will need.

Current status

Currently the project is between 50% and 60% on the basic "scooter" step. Here the goals were:

  1. Create a functional NUOPC cap for GEOS which would allow the correct fields to make it to history (complete)
  2. Create a History Writer which could write a collection to a netcdf file using MAPL's PFIO utilities.
  3. Create the start of a new yaml format to configure the NUOPC cap and writer (complete).
  4. Create a standalone test program to test the current state of History TNG (partially complete).

Overview of Current state of History TNG code

All history TNG code is currently contained on the feature/wjamieson/History_TNG branch of MAPL. The majority of the new code is contained in the gridcomps/History_TNG directory. The exceptions to this are the components of History TNG which require changes to the MAPL cap (the NUOPC cap), meaning those parts are contained within the gridcomps/Cap](https://github.com/GEOS-ESM/MAPL/tree/feature/wjamieson/History_TNG/gridcomps/Cap) directory.

Contents of History_TNG Directory

In the History_TNG directory contains all the major objects that keep track of the collection configurations. In particular, I have taken care to carefully encapsulate all of the information needed by a history component so that in the future features should be both simple and have a natural home. Moreover, this approach allows the possibility of code reuse via inheritance with minimal refactoring.

The class structure of History TNG is nested starting with handling individual fields all the way through handling sets of collections. The majority of the initial work for History TNG involves the introduction of NUOPC, which requires us to figure out how to pass individual fields from MAPL to NUOPC. This has lead to an initial focus on carefully handling how History TNG stores information on the fields it needs to work with.

History TNG's handling of Individual Fields

For the "scooter" version of History TNG, it was extremely important to describe how to handle each field the History TNG will need to deal with. This is so that the NUOPC cap has enough information so that it can reach into GEOS through MAPL and extract the necessary fields for History TNG to then write to files and likewise have enough information for the History Writer to tell NUOPC which fields it needs passed to it. The primary class for describing this is the FieldEntry class.

FieldEntry requires two primary pieces of information for a given field, namely:

  1. The MAPL short name
  2. The MAPL component name

These are needed so that the NUOPC cap and MAPL can find the correct fields in MAPL in order to pass them up into NUOPC. It has more configurable information, so that one could configure the NUOPC configuration, namely:

  1. The field units (NUOPC requires that one registers the units for a field, and at the current time it is easier to let the user specify these or allow the default NUOPC unit than it was to derive this information from MAPL. Doing so was on my TODO list for future versions of History TNG. Moreover, allowing the user to specify units could allow for automatic unit conversions in History TNG.) Currently, this defaults to "1" if not specified.
  2. TransferOfferGeomObject This is a NUOPC option which allows components to agree on which component or components can specify the geometry (grid) that the field uses so that NUOPC can attempt to use its automatic re-gridding capability. Currently, this defaults to the NUOPC default which is "will provide".
  3. SharePolicyField This is a NUOPC option which allows for direct pointer sharing between components. Currently, this defaults to the NUOPC default which is "not share".
  4. SharePolicyGeomObject This is a NUOPC option which allows a field to set the geometry (grid) for another NUOPC component. Currently, this defaults to the NUOPC default which is the value SharePolicyField is set to.

Note that the NUOPC configuration options are how NUOPC negotiates among components to allow for fields to be transferred. The NUOPC defaults will work for the current "scooter" version, but they may need to be tweaked for later versions of History TNG. Moreover, a more general NUOPC cap for all of GEOS will need to be able to individually configure all these options.

FieldEntry contains all the methods needed to handle a single field with History TNG such as the necessary advertise and realize steps that NUOPC needs to perform in addition to other book-keeping tasks. In the future, I expect that per-field history configuration options would be implemented at this level.

The FieldEntryMap is simply a gFTL string-FieldEntry map.

For the purposes of the NUOPC cap, I needed to introduce a collection of all fields that History TNG will need to write across all active collections. Thus I wrapped a FieldEntryMap inside a class to form the FieldRegistry. The field registry handles all the configurable portions of History TNG which requires special behavior by the NUOPC cap. This primarily is advertising and realizing all the fields that GEOS needs to export into NUOPC for the current history configuration.

Field Groups

@tclune noted that History really needed a notion of sets of fields which could be configured in different ways for different collections, in order to reduce the duplication of information in the configuration file. This lead to the notion of a Group, which is a set of fields that one can define together.

In order to implement this I need to break the it into a few smaller parts. First, we needed a FieldGroupEntry which wraps the FieldEntry class together with the additional information that the History writer will need to know about the field, in this case the field's optional alias_name. However, more information can be included at this point. The [FieldGroupEntryMap](https://github.com/GEOS-ESM/MAPL/blob/feature/wjamieson/History_TNG/gridcomps/History_TNG/FieldGroupEntryMap.F90

Clone this wiki locally