load esmf/8.5.0 in gaea.intel.lua

[RussTreadon-NOAA]

This PR adds load("esmf/8.5.0") to modulefiles/GDAS/gaea.intel.lua. This is required for the GDASApp build to successfully complete on Gaea C5.

Resolves #1326

[RussTreadon-NOAA]

I don't think the automatic check failures are due to the changes in this PR. There have been several GitHub Incident - Disruption with some GitHub services - 11 October 2024 notifications within the last hour.

[RussTreadon-NOAA]

Confirmed that adding load("esmf/8.5.0") to gaea.intel.lua is sufficient to allow build.sh to successfully run to completion on Gaea C5. Test build completed in /gpfs/f5/nggps_emc/scratch/Russ.Treadon/git/GDASApp/test.

[CoryMartin-NOAA]

@RussTreadon-NOAA sorry I didn't catch this sooner, but I see that C5 is having an OS upgrade next week... I fear this will make a lot of spack-stack things become broken, so perhaps it's best to pause this until after that maintenance/upgrade?

[RussTreadon-NOAA]

Oops, that's unfortunate. I'll flip this PR to draft since what works today will likely fail next week.

[CoryMartin-NOAA]

I just so happened to catch "Gaea C5 upgrade" on my calendar since I subscribe to the RDHPCS calendar. Email search says it is for an OS upgrade. If it is anything like the upgrade on other machines, it will be quite disruptive.

[RussTreadon-NOAA]

Agreed. This PR may be in draft mode for quite some time. If the wait gets too long, we can simply close this PR and open a new one once the OS upgrade dust settles.

[RussTreadon-NOAA]

Successfully build GDASApp on Gaea C5 using feature/gaea-c5 at a140bbd. Given this, mark this PR as Ready for review.

[RussTreadon-NOAA]

GDASApp ctests

Unfortunately several GDASApp ctests fail on Gaea C5

est project /gpfs/f5/nggps_emc/scratch/Russ.Treadon/git/GDASApp/test/build
      Start 1579: test_gdasapp_util_coding_norms
 1/23 Test #1579: test_gdasapp_util_coding_norms ..........   Passed    4.09 sec
      Start 1580: test_gdasapp_util_ioda_example
 2/23 Test #1580: test_gdasapp_util_ioda_example ..........   Passed    0.65 sec
      Start 1581: test_gdasapp_util_prepdata
 3/23 Test #1581: test_gdasapp_util_prepdata ..............***Failed    0.38 sec
      Start 1582: test_gdasapp_util_rads2ioda
 4/23 Test #1582: test_gdasapp_util_rads2ioda .............   Passed    0.36 sec
      Start 1583: test_gdasapp_util_ghrsst2ioda
 5/23 Test #1583: test_gdasapp_util_ghrsst2ioda ...........Subprocess aborted***Exception:   0.42 sec

...

      Start 1949: test_gdasapp_convert_bufr_adpsfc_snow
21/23 Test #1949: test_gdasapp_convert_bufr_adpsfc_snow ...   Passed    4.08 sec
      Start 1950: test_gdasapp_convert_bufr_adpsfc
22/23 Test #1950: test_gdasapp_convert_bufr_adpsfc ........   Passed    4.62 sec
      Start 1951: test_gdasapp_convert_gsi_satbias
23/23 Test #1951: test_gdasapp_convert_gsi_satbias ........   Passed   18.00 sec

39% tests passed, 14 tests failed out of 23

Label Time Summary:
gdas-utils    =   8.29 sec*proc (12 tests)
script        =   8.29 sec*proc (12 tests)

Total Test time (real) =  49.75 sec

The following tests FAILED:
        1581 - test_gdasapp_util_prepdata (Failed)
        1583 - test_gdasapp_util_ghrsst2ioda (Subprocess aborted)
        1584 - test_gdasapp_util_smap2ioda (Subprocess aborted)
        1585 - test_gdasapp_util_smos2ioda (Subprocess aborted)
        1586 - test_gdasapp_util_viirsaod2ioda (Subprocess aborted)
        1587 - test_gdasapp_util_icecabi2ioda (Subprocess aborted)
        1588 - test_gdasapp_util_icecamsr2ioda (Subprocess aborted)
        1589 - test_gdasapp_util_icecmirs2ioda (Subprocess aborted)
        1590 - test_gdasapp_util_icecjpssrr2ioda (Subprocess aborted)
        1944 - test_gdasapp_fv3jedi_fv3inc (Failed)
        1945 - test_gdasapp_snow_create_ens (Failed)
        1946 - test_gdasapp_snow_imsproc (Failed)
        1947 - test_gdasapp_snow_apply_jediincr (Failed)
        1948 - test_gdasapp_snow_letkfoi_snowda (Failed)
Errors while running CTest

[RussTreadon-NOAA]

Rerun test_gdasapp_util_ghrsst2ioda with -VV. Output contains

1583: Test command: /gpfs/f5/nggps_emc/scratch/Russ.Treadon/git/GDASApp/test/build/bin/gdas_obsprovider2ioda.x "../testinput/gdas_ghrsst2ioda.yaml"
1583: Environment variables:
1583:  OMP_NUM_THREADS=1
1583: Test timeout computed to be: 1500
1583: OOPS Starting 2024-10-24 13:15:13 (UTC-0400)
1583: [TestReference] Comparing to reference file: testref/ghrsst2ioda.test
1583: Relative float tolerance for tests : 1e-06
1583: Absolute float tolerance for tests : 0
1583: [TestReference] Saving Test output to: testoutput/ghrsst2ioda.test
1583: Configuration input file is: ../testinput/gdas_ghrsst2ioda.yaml
1583: Full configuration is:YAMLConfiguration[path=../testinput/gdas_ghrsst2ioda.yaml, root={provider => GHRSST , window begin => 2021-03-24T15:00:00Z , window end => 2021-03\
-24T21:00:00Z , binning => {stride => 2 , min number of obs => 1} , bounds => {min => -3 , max => 50} , output file => ghrsst_sst_ma_20210324.ioda.nc , input files => (ghrsst\
_sst_ma_202103241540.nc4,ghrsst_sst_ma_202103241550.nc4) , test => {reference filename => testref/ghrsst2ioda.test , test output filename => testoutput/ghrsst2ioda.test , flo\
at relative tolerance => 1e-06}}]
1583: OOPS_STATS ObjectCountHelper started.
1583: OOPS_STATS Run start                                - Runtime:      1.01 sec,  Memory: total:    60.92 Mb, per task: min =    60.92 Mb, max =    60.92 Mb
1583: Run: Starting gdasapp::ObsProvider2IodaApp
1583: --- Window begin: 2021-03-24T15:00:00Z
1583: --- Window end: 2021-03-24T21:00:00Z
1583: --- Input files: [ghrsst_sst_ma_202103241540.nc4,ghrsst_sst_ma_202103241550.nc4]
1583: --- Output files: ghrsst_sst_ma_20210324.ioda.nc
1583: Processing files provided by GHRSST
1583: Exception: No such file or directory
1583: file: ncFile.cpp  line:88
1583: Exception: gdasapp::ObsProvider2IodaApp terminating...
1583: Exception: level: 0
1583: No such file or directory
1583: file: ncFile.cpp  line:88
1583: terminate called after throwing an instance of 'oops::TestReferenceMissingTestLineError'
1583:   what():  TestReference: Missing test output line corresponding to reference file Line#:1
1583: Ref line: 'Reading: [ghrsst_sst_ma_202103241540.nc4,ghrsst_sst_ma_202103241550.nc4]'
1/1 Test #1583: test_gdasapp_util_ghrsst2ioda ....Subprocess aborted***Exception:   7.88 sec

0% tests passed, 1 tests failed out of 1

@DavidNew-NOAA , is this failure an example of what you mentioned during today's (10/24) JEDI-T2O meeting: develop: works for non-marine DA apps? Do we have a tentative fix?

[CoryMartin-NOAA]

@RussTreadon-NOAA I thought that was in reference to the gdas.cd hash update PR that, while breaking aero and snow, fixes soca?