Skip to content

LDAS Regression Tests

Jump to bottom
Rolf Reichle edited this page Aug 27, 2024 · 2 revisions

Introduction

The LDAS Regression Tests (formerly "Nightly" Tests) are a set of short, self-contained GEOSldas runs that mimic important applications (e.g., SMAP L4_SM). The tests are implemented at NCCS only.

Objectives:

  1. For GEOSldas development, the modified and original source code should generate test run output that is binary identical, if at all possible. If so, the modifications are considered "zero-diff" (see Pull Requests).

  2. The test runs provide examples of GEOSldas configurations (skip to Step 2 below).

Any changes in the GEOSldas, GEOSldas_GridComp, or GEOSgcm_GridComp repositories must be tested because GEOSldas includes the GEOSldas_GridComp and GEOSgcm_GridComp repositories via GEOSldas/components.yaml.

The reference runs for the tests use the respective develop branches.

The LDAS tests also check for differences in model diagnostic output (incl. netcdf, binary, and text files), in addition to differences in restart/checkpoint files.

Suppose you have updates in one or more of the following branches:

  1. b_mygeosldas for GEOSldas
  2. b_mygeosldasgridcomp for GEOSldas_GridComp
  3. b_mygeosgcmgridcomp for GEOSgcm_GridComp

Step 1: Push branch[es] to GitHub

Make sure b_mygeosldas, b_mygeosldasgridcomp, and b_mygeosgcmgridcomp have been pushed to GitHub.

If your modifications only impact GEOSldas, make sure GEOSldas/components.yaml points to the "develop" branches of GEOSldas_GridComp and GEOSgcm_GridComp.

If your modifications only impact GEOSldas_GridComp and/or GEOSgcm_GridComp:

  1. Create a new GEOSldas branch b_mygeosldas off the develop branch.
  2. On the new branch, edit GEOSldas/components.yaml and point GEOSldas_GridComp and/or GEOSgcm_GridComp to your branches b_mygeosldasgridcomp and/or b_mygeosgcmgridcomp as follows:
[GEOSldas_GridComp]
...
branch = b_mygeosldasgridcomp
...
[GEOSgcm_GridComp]
...
branch = b_mygeosgcmgridcomp
  1. Commit and push to GitHub.

Step 2: Run Test Script

Ssh to a SLES15 login node: ssh discover-mil. [NEW AS OF 27 AUG 2024!!!]

Make sure nccmp is in your PATH: module load nccmp.

Then run the following commands:

cd /discover/home/mathomp4/SI_Team/scripts/systest/

./LDAS_Regression.csh -b [branch] -e [email] -s [sponsor] -t conus -t global -t globalcs -t globalcnclm4

where [branch] is your GEOSldas branch (e.g., b_mygeosldas), [email] is your email address, and [sponsor] is your NCCS sponsor code (e.g., g0610). The -t arguments specify the specific set of tests that must be conducted by developers (see below for details).

IMPORTANT: The LDAS_Regression.csh script can be executed with a subset of these tests, but -t conus must always be selected as the first test because the script re-uses the checkout and build generated during the "conus" test.

To generate GEOSldas sample configurations, specify develop or a tag/release name as your [branch]. The tests are maintained to work and "pass" for the develop branch. For an older tag/release, the tests may "fail" the comparison if there were non-zero-diff changes, or they may not work at all if configuration details have changed. There may also be short periods when the tests fail the output comparison even for develop or do not work at all because of ongoing maintenance of the tests.

Test results

Upon completion, the script will send an email to the address specified above with a summary of the test results. A typical test summary is shown here:

Full Log File: /discover/nobackup/[username]/SystemTests/logs/LDAS/[YYYY-MM-DD]/log.[HH:MM:SS]

Runtype         Clone Build Build Time Model Run/Compare Assim Run/Compare
-------         ----- ----- ---------- ----------------- -----------------
conus           pass  pass  17 min     pass/pass          -- / --         
global           --    --    --        pass/pass         pass/pass        
globalcs         --    --    --        pass/pass         pass/pass        
globalcnclm4     --    --    --        pass/pass          -- / --

See the "Full Log File" for additional information.

If 'FAIL' appears anywhere, the modifications are not zero-diff. Nonzero-diff results, even when unavoidable or intentional, require special approval from the modeling and land assimilation leads before they can be merged (see Pull Requests).

Occasionally, tests may fail because of issues with Discover or because the team is in the process of updating the test baseline. If you think this may happening, please check with the gatekeepers or the GMAO software infrastructure team.

Suite of LDAS tests

The tests cover different aspects of GEOSldas:

  • "conus" is a Catchment model simulation over the CONUS domain in the EASEv2 tile space.
  • "global" is a global Catchment model simulation in the EASEv2 tile space, with the "assimilation" case testing GEOSldas as used in SMAP L4_SM.
  • "globalcs" is a global Catchment model simulation and an assimilation case in the cube-sphere tile space.
  • "globalcnclm4" is a global CatchmentCNCLM4 model simulation in the EASEv2 tile space.

Additional LDAS tests with debug, aggressive, and/or GNU-compiler builds are available (see /discover/nobackup/mathomp4/SystemTests/runs/).

GEOSldas Wiki Home

GEOSldas Wiki Home

Clone this wiki locally