Add TimeSeriesStatsRestartTest to global ocean

[xylar]

This merge adds a new TimeSeriesStatsRestartTest test case that makes sure the timeSeriesStats* analysis member
produces the same results with a full 8-hour run and with two 4-hour runs with a restart in between. Tests are available with or without analysis restart files (the former is the MPAS-Ocean standalone default, the latter is the behavior in E3SM).

These tests exist with the variables used in both daily and monthly averages, though they are not computed over so long a time interval for efficiency.

QU240 versions of these test cases have been added to the pr test suite for both daily and monthly output and both with and without analysis restarts.

The tests will fail with the current E3SM master. They pass with the bug fixes in E3SM-Project/E3SM#5218.

[mark-petersen]

Tested with gnu debug, gnu optimized, and intel optimized using all daily and monthly timeSeriesStats tests and this PR for compass, and the code PR E3SM-Project/E3SM#5218.

I temporarily added a test to compass that could be added to this PR, if you think it would be useful @xylar

cat ./compass/ocean/suites/timeseriesstats.txt

ocean/global_ocean/QU240/mesh
ocean/global_ocean/QU240/PHC/init
ocean/global_ocean/QU240/PHC/daily_output_test
ocean/global_ocean/QU240/PHC/daily_analysis_restart
ocean/global_ocean/QU240/PHC/daily_model_restart
ocean/global_ocean/QU240/PHC/monthly_output_test
ocean/global_ocean/QU240/PHC/monthly_analysis_restart
ocean/global_ocean/QU240/PHC/monthly_model_restart

[xylar]

@mark-petersen, thanks very much for testing! I added the suite as you suggested. To be consistent with other suite names, I used underscores in the name: time_series_stats. I hope that's okay with you.

[xylar]

Testing

I ran the pr test suite on Anvil with Intel and IMPI using this branch and E3SM-Project/E3SM#5202 as a baseline, and this branch an E3SM-Project/E3SM#5218 for comparison. All tests ran successfully and most validation failures were as expected:

• the new tests failed validation when using Add CF-compliant Time coordinate E3SM-Project/E3SM#5202, as expected (that's what Fix two bugs in timeSeriesStats output in MPAS-Ocean E3SM-Project/E3SM#5218 is designed to fix)
• the new tests passed internal validation but failed baseline validation (because the baseline restarts are broken, again what Fix two bugs in timeSeriesStats output in MPAS-Ocean E3SM-Project/E3SM#5218 is fixing).

However, I saw validation failures in both suits for ocean_baroclinic_channel_10km_decomp_test. I can't imagine that is related to this PR but it should be investigated. It seems likely to have been introduced by some other recent E3SM PR.

[mark-petersen]

@xylar thanks. Yes, underscores are better. What compiler and machine did you see the failure, and debug or optimized? I also saw a QU240 decomp failure but only with intel optimized. I did not see any with the baroclinic channel, testing master today.

[xylar]

@mark-petersen, as I said above, Anvil, Intel, Intel-MPI. I was running optimized.

[xylar]

I can use #354 to find out which E3SM merge introduced this.

[mark-petersen]

See issue on test failure at E3SM-Project/E3SM#5219

[xylar]

I am going to close this for now, as no progress has been made in more than a year.