Dummy PR to trigger CI #2938
Closed
Dummy PR to trigger CI #2938
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updates ufs-weather-model, this updates RDHPCS to the newer spack-stack allowing some temporary fixes to be reverted. * removes upp submodule * uses upp from the ufs-weather-model * restores the build and link that were hacked during the Hera Rocky 8 transition to allow for UPP submodule * Removes forecast directories in clean-up Resolves NOAA-EMC#2617 Resolves NOAA-EMC#2437 --------- Co-authored-by: Rahul Mahajan <[email protected]>
…OAA-EMC#2632) This PR enables linking both global-nest fix files and non-nest ones at the same time and users can run both nesting and non-nesting experiments at the same time without worries about what fix files to be linked. Resolves NOAA-EMC#2631
Adds a COM template to define a path to store obs processed for JEDI. This will allow UFSDA to stop writing to COM_OBS, which should be read-only as it belongs to obsproc in operations. No functional change yet.
This PR addresses issue NOAA-EMC#2444. The following is accomplished: - All `sleep` statements are replaced with `wait_for_file` for the relevant scripts beneath `scripts` and `ush`; - Indentation and shell-norms are updated where applicable. Note: The WAFS scripts are not updated as per @aerorahul direction. Resolves NOAA-EMC#2444 --------- Co-authored-by: henrywinterbottom-wxdev <[email protected]> Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
The PR allows the use of ICs from PSL's replay analysis. These replay ICs will be used for GEFS reforecasting and SFS. Two main changes are associated with these updates: (1) replay ICs being valid at 3Z, and (2) the use of warm starts. Resolves NOAA-EMC#1838 --------- Co-authored-by: Jessica Meixner <[email protected]> Co-authored-by: Walter Kolczynski - NOAA <[email protected]> Co-authored-by: Rahul Mahajan <[email protected]>
This PR updates the `sorc/gdas.cd` and `sorc/gsi_utils` hashes. The updated hashes bring in bug fixes, new UFS DA functionality, and a Gaea build for gsi_utils. Resolves NOAA-EMC#2640
…#2685) This quick-fix PR is to update the Jenkins Pipeline's configurable parameter list to include the **Hercules-EMC** node. This allows Jenkins users to restart Jobs in the controller when no updates have been made.
needed a comma
This PR removes the ocn_daily files that are produced by the ocean component. These files can be recreated by averaging data that exists in the 6 hour aveaged files if needed. Fixes NOAA-EMC#2675 Fixes NOAA-EMC#2659 (by removing them and making this obsolete)
…#2624) Add a prepaeroobs job to prepare aerosol obs files for DA. This job does quality control of the VIIRS aerosol raw observations and convert them to ioda format. Resolves NOAA-EMC#2623 --------- Co-authored-by: ypwang19 <[email protected]> Co-authored-by: TerrenceMcGuinness-NOAA <[email protected]> Co-authored-by: Cory Martin <[email protected]> Co-authored-by: David Huber <[email protected]>
This PR moves JCB into GDASapp. The PR also bumps up the hash of GDASapp to what is in `feature/move_jcb`, which at time of writing is develop plus the absorption of JCB into GDASapp. Note that I also took the changes from NOAA-EMC#2641 to follow the testing @RussTreadon-NOAA has done.
Optimize the gfswavepostpnt, gfswavepostbndpntbll, and gfswavepostbndpnt jobs
This is done by
1) reducing the number of calls to `sed`, `awk`, `grep`, and `cat` by
- performing operations on all files at once instead of looping over
each file
- removing piped `cat` calls (e.g. `cat <file> | sed 'something'`)
- combining `sed` and `grep` calls when possible
- adding logic to `awk` calls instead of handling that logic in bash
2) minimizing as much as possible the amount of data on disk that has to
be read in (e.g. limiting sed to read only the line numbers it needs)
---------
Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
Turns on high-frequency (hourly) output in the extended products test to exercise that aspect of the code. This test only runs on WCOSS. Also adds the hooks to optionally turn on the metplus jobs, but that is deferred as they are not currently working correctly.
NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow archive task. Refs NOAA-EMC#2451 --------- Co-authored-by: Walter Kolczynski - NOAA <[email protected]> Co-authored-by: Rahul Mahajan <[email protected]> Co-authored-by: David Huber <[email protected]>
This PR is a hotfix for an incorrectly named (e.g., non-existent) `COM/` template. Resolves NOAA-EMC#2696 Refs NOAA-EMC#2451
…put files (NOAA-EMC#2688) This PR addresses issue NOAA-EMC#1252. The following is accomplished: - Prior to removing files, the existence of a file is checked prior to attempting to remove; this is performed as noted [here](NOAA-EMC#1252 (comment)); this PR only addresses the the `chgrp` issue. Refs NOAA-EMC#1252 --------- Co-authored-by: David Huber <[email protected]>
Eliminates the post groups used for upp and products jobs so that each task only processes one forecast hour. This is more efficient and greatly simplifies downstream dependencies that depend on a specific forecast hour. Resolves NOAA-EMC#2666 Refs NOAA-EMC#2642
…#2687) This fixes the online archiving portion of the `*arch` and `*earc00` jobs, a prerequisite for running METplus. This also reenables METplus by default. The approach previously taken created `FileHandler` dictionaries at varying levels within the resulting yaml, which was not properly parsed by `exglobal_archive.py`. This approach creates a single `FileHandler` dictionary and is much less complicated overall. Resolves NOAA-EMC#2673 NOAA-EMC#2647
Adds a table to HPC documentation stating the minimum support versions
Redefine resource variables based explicitly on RUN or CDUMP
Additionally, machine-specific resources are moved out of
config.resources and placed in respective config.resources.{machine}
files.
Resolves NOAA-EMC#177 NOAA-EMC#2672
NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow wave model and products tasks. Refs NOAA-EMC#2451
Updates the logic to be by run instead of for DO_JEDIOCNVAR to determine how many layers and sets the exception value for MOM6 to be 1e-34 for all scenarios. Note, we will no longer have zeros in the ocean grib output and the DA will also run without issues. Fixes NOAA-EMC#2615
This PR: - removes stale temporary scratch run directories from `$DATAROOT/` every 3 days. - should help to scrub failed attempts. - removes an unused variable `RUNDIR` defined in `config.base`
Activate the snow DA test on WCOSS.
…ut (NOAA-EMC#2692) We are updating the Jenkins Pipeline with a try block around checkout to capture errors for the user. Also cleaned up Finalize and added section to clean out workspace on success.
This PR: - reverts NOAA-EMC#2681 in part - keeps some changes for `RUN`. - is a hotfix - should be merged ASAP after consensus w/ @guillaumevernieres @CatherineThomas-NOAA @WalterKolczynski-NOAA
…OAA-EMC#2700) This PR updates the `gdas.cd` hash to bring in new JCB conventions. Resolves NOAA-EMC#2699 From NOAA-EMC#2654 This PR will move much of the staging code that take place in the python initialization subroutines of the variational and ensemble DA jobs into Jinja2-templated YAML files to be passed into the wxflow file handler. Much of the staging has already been done this way, but this PR simply expands that strategy. The old Python routines that were doing this staging are now removed. This is part of a broader refactoring of the pygfs tasking. wxflow PR [NOAA-EMC#30](NOAA-EMC/wxflow#30) is a companion to this PR. Co-authored-by: danholdaway <[email protected]> Co-authored-by: DavidNew-NOAA <[email protected]> Co-authored-by: DavidNew-NOAA <[email protected]> Co-authored-by: Dan Holdaway <[email protected]>
…OAA-EMC#2701) This PR sets up the ability on Gaea for auto generation of a clean xml file, i.e., an xml file that does not need any alterations before running rocoto. Refs NOAA-EMC#2572 Refs NOAA-EMC#2664
Fixes an issue created from NOAA-EMC#2701 that added `CLUSTERS` to the `gaea.yaml`.
This PR updates gempak jobs (gfs, gdas, and goes) from processing all forecast hours at once to one forecast hour at a time. This will reduce the job runtime to less than 5 min, so restart capability is not needed. Resolves NOAA-EMC#1250 Ref NOAA-EMC#2666 NOAA-EMC#2667 --------- Co-authored-by: Walter.Kolczynski <[email protected]>
This PR adds JEDI ATM lgetkf observer and solver jobs to global-workflow. This is approach is akin GSI-based eobs and eupd. Splitting the single JEDI ATM lgetkf job into separate observer and solver jobs improves memory and computational efficiency. Resolves NOAA-EMC#2415
Make ATM-OCN-ICE coupling model run on AWS. This adds capability to run UFS atm-ocn-ice coupling on AWS. Resolves NOAA-EMC#2858
This PR corrects a bug in the staging job for ocean `MOM.res_#` IC files. The `OCNRES` value was coming in as an integer (e.g. `25`) but the `ocean.yaml.j2` file was checking for `"025"`. Correct to now set OCNRES to be three digits in staging script and also correct the for loop range to include third file. Resolves NOAA-EMC#2864
…AA-EMC#2738) This PR adds in support for computing files needed for the aerosol analysis **B**. This includes a new task, `aeroanlgenb`. This work was performed by both me and @andytangborn Resolves NOAA-EMC#2501 Resolves NOAA-EMC#2737 --------- Co-authored-by: Andrew.Tangborn <[email protected]> Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
Adds files `atmi009.nc`, `atmi003.nc`, `ratmi009.nc`, and `ratmi003.nc` to list of files to be staged for ICs, if available. These are necessary for starting an IAU run, and are currently missing. Resolves NOAA-EMC#2874
# Description Support global-worflow GEFS C48 on Google Cloud. Make env. var. and yaml file changes, so global-workflow GEFS C48 case can run properly on Google Cloud. Resolves NOAA-EMC#2860
Add the ability to run CI test C96_atm3DVar on Gaea-C5 Resolves NOAA-EMC#2766 Refs NOAA-EMC/prepobs#32 Refs NOAA-EMC/Fit2Obs#28
Use the updated 2013 to 2024 mean MERRA2 climatology instead of 2003 to 2014 mean Depends on NOAA-EMC#2887 Refs: ufs-community/ufs-weather-model#2272 Refs: ufs-community/ufs-weather-model#2273
…MC#2893) This changes the order of the cleanup job so that the working directory is deleted at the end. It also adds the `-ignore_readdir_race` flag to `find` to prevent errors if a file was deleted after the list of files was collected. This can happen if two consecutive cycles run the cleanup job at the same time.
This updates the model hash to include the UPP update needed to be able to run the post processor on Orion, thus reenabling support on that system. A note on the UPP: it is using a newer version of g2tmpl that requires a separate spack-stack 1.6.0 installation. This version of g2tmpl will be standard in spack-stack 1.8.0, but for now requires loading separate modules for the UPP. A note on running analyses on Orion: due to a yet-unknown issue causing the BUFR library to run much slower on Orion when compared with Rocky 8, the GSI and GDASApp are expected to run significantly slower than on any other platform (on the order of an hour longer). Lastly, I made adjustments to the build_all.sh script to send more cores to compiling the UFS and GDASApp. Under this configuration, the GSI, UPP, UFS_Utils, and WW3 pre/post executables finish compiling before the UFS when run with 20 cores. Resolves NOAA-EMC#2694 Resolves NOAA-EMC#2851 --------- Co-authored-by: Rahul Mahajan <[email protected]> Co-authored-by: Walter.Kolczynski <[email protected]>
…#2816) - This task is an extension of the empty arch job previously merged. - This feature adds an archive task to GEFS system to archive files locally. - This feature archives files in ensstat directory. Resolves NOAA-EMC#2698 Refs NOAA-EMC#832 NOAA-EMC#2772
The current operational BUFR job begins concurrently with the GFS model run. This PR updates the script and ush to process all forecast hour data simultaneously, then combines the temporary outputs to create BUFR sounding products for each station. The updated job will now start processing data only after the GFS model completes its 180-hour run, handling all forecast files from 000hr to 180hr at a time. The new version job running will need 7 nodes instead of the current operational 4 nodes. This PR depends on the GFS bufr code update NOAA-EMC/gfs-utils#75 With the updates of bufr codes and scripts, there is no need to add restart capability to GFS post-process job JGFS_ATMOS_POSTSND. This PR includes the other changes: Rename the following table files: parm/product/bufr_ij13km.txt to parm/product/bufr_ij_gfs_C768.txt parm/product/bufr_ij9km.txt to parm/product/bufr_ij_gfs_C1152.txt Add a new table file: parm/product/bufr_ij_gfs_C96.txt for GFSv17 C96 testing. Added a new capability to the BUFR package. The job priority is to read bufr_ij_gfs_${CASE}.txt. If the table file is not available, the code will automatically find the nearest neighbor grid point (i, j). Refs NOAA-EMC#1257 Refs NOAA-EMC/gfs-utils#75
This PR creates a PyGFS class called JEDI, which is to be instantiated everytime a JEDI application is run. The AtmAnalysis and AtmEnsAnalysis classes are no longer children of the Analysis class, but rather direct children of the Task class. They each have a JEDI object as an attribute, which is used to run either the variational/ensemble DA JEDI applications or the FV3 increment converter JEDI application, depending on which job they are created for (e.g. atmanlvar vs. atmanlfv3inc). The intention is that a later PR will apply this framework to all analysis task, and the PyGFS Analysis class will be removed.
This PR: - Creates a standalone page for FAQ and Common issues - Adds a block of caution on using variables in a users' `bashrc` Fixes: NOAA-EMC#2850
This modifies the way the `config` dictionary is constructed and referenced. Rather than updating a single configuration dictionary with each `RUN`, a `RUN`-based dictionary of `config` dictionaries is created and referenced by the appropriate `RUN` when calculating resources. This also makes the methods that were hidden before NOAA-EMC#2727 hidden again. Resolves NOAA-EMC#2783
This replaces `APRUN` with `APRUN_default` in all of the `.env` files. Resolves NOAA-EMC#2870
This adds 3 missing links from the UPP into parm/ufs to .gitignore. Resolves NOAA-EMC#2901
…ression options (NOAA-EMC#2914) - enables writing native grid model output when doing JEDI-atm DA - updates compression settings for C384 model output Fixes NOAA-EMC#2891
Support global-worflow GEFS C48 on Azure Make env. var. and yaml file changes, so global-workflow GEFS C48 case can run properly on Google Cloud. Resolves NOAA-EMC#2882
…C#2922) Adds 1 deg ocean, ice information to config.resources so 1 deg ocean jobs can run
In preparation for GDASApp CI tests, add GDASApp build capability to global-workflow and remove memory specifications for Gaea-C5 xml setup (in Ref to NOAA-EMC#2727) Resolves NOAA-EMC#2535 Resolves NOAA-EMC#2910 Resolves NOAA-EMC#2911
As forecast ensemble jobs are added to the global workflow, this PR ensures the output is being cleaned up properly once it is no longer needed. Resolves NOAA-EMC#833
This PR updates the parm/config/gfs/config.resources and env/WCOSS2.env files for the BUFR sounding job "postsnd." It includes adjustments to resource settings such as tasks per node and memory allocations for various GFS resolutions, including C768, C1152, and others. Here are the proposed changes: C768: 7 nodes, 21 tasks per node C1152: 16 nodes, 9 tasks per node
NCO has requested that each COM variable specify whether it is an input or an output. This completes that process for the global-workflow Unified Post Processor (UPP) task. Refs: NOAA-EMC#2451
This PR updates the `develop` branch to use the newer operational
`obsproc/v1.2.0` and `prepobs/v1.1.0`. The obsproc/prepobs installs in
glopara space on supported platforms use tags cut from the `dev/gfsv17`
branches in the respective repos. The installation of `prepobs/v1.1.0`
on WCOSS2 is called "gfsv17_v1.1.0" to help avoid GFSv16 users using it
instead of the operational module.
Also, the `HOMEobsproc` path is updated to set an empty default for
`obsproc_run_ver`. This both removes the need to set a default (and
constantly update it, which is duplication) and avoid the unset variable
error when the fcst jobs use their own load module script that does not
know `obsproc_run_ver`:
```
export HOMEobsproc="${BASE_GIT:-}/obsproc/v${obsproc_run_ver:-}"
```
This PR also reverts the prepobs and fit2obs installs on MSU back to the
glopara space from the temporary `/work/noaa/global/kfriedma/glopara`
space installs.
Lastly, this PR also includes updates to complete issue NOAA-EMC#2844 (merge
`build.spack.ver` and `run.spack.ver`).
Resolves NOAA-EMC#2291
Resolves NOAA-EMC#2840
Resolves NOAA-EMC#2844
|
I am opening new PR. something went wrong. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Type of change
Change characteristics
How has this been tested?
Checklist