Skip to content

CAM SE Workflows

Jump to bottom
cacraigucar edited this page Sep 19, 2024 · 29 revisions

CAM SE Workflows

This page describes the process once a scientist has indicated on their issue that their branch is ready for cam_development. This documents the steps that the assigned SE will perform.

Table of Contents

CAM SE scheduler duties

  1. Enter a card for the request onto the CAM project board (barring PTO this should happen within a day of the request)
  2. If possible, assign the SE who will be responsible for bringing this onto cam_development and/or a release branch.

Primary Assigned SE's duties for when code from scientists are being brought into CAM

IF SCIENTIST HAS OPENED A PR

  1. Discuss with scientist if they want:

    a. You push your changes back to their branch in the PR directly - then you clone their branch directly and push changes back to it

    b. You issue a PR back to their branch in the PR - you create a branch in your fork using their branch in the PR and when edits are complete you issue a PR back to their branch.

    c. You work exclusively in a branch on your fork (the merge request will come from you fork/branch). In this case make sure you add the text "Closes #XXX" where XXX is the number of the PR in the scientist's branch.

  2. Perform necessary cleanup (if any needed) prior to requesting additional CAM SE reviewers. In particular, you should update your branch to the head of the PR target (cam_development or release branch).

  3. Consider if a new CAM regression test needs to be added. If so, discuss with the scientist about getting an appropriate test or modifying an existing test.

  4. Run the CAM Tests

  5. Get the final edits back into the scientist's PR

  6. Hit the merge button

NOTE -- The steps above have more details in the steps below.

IF EXTENSIVE MODIFICATIONS ARE REQUIRED OR NO PR HAS BEEN OPENED - this will involve opening a new PR

  1. Copy scientist branch to your fork

  2. Perform necessary cleanup (if any needed) prior to issuing a pull request. In particular, you should update your branch to the head of the PR target (cam_development or release branch).

  3. Consider if a new CAM regression test needs to be added. If so, discuss with the scientist about getting an appropriate test or modifying an existing test.

  4. Run the CAM Tests

  5. Go to your CAM fork's GitHub page, and select the "Pull requests" tab:

  6. Create a new pull request using the "new pull request" button:

  7. Set the "base repository" to "ESCOMP/CAM", and the "base" to your branch of interest (likely "cam_development"):

  8. Set the "head repository" to your CAM fork repo, and the "compare" to your newly created branch:

  9. If the phrase "Able to merge" is present, then select "create pull request":

  10. Add a title for your request, along with a description of what your pull request is doing. Fill out all fields in the template while removing any template comments. Be sure to list the issue(s) which are addressed by this pull request, taking care to use the correct syntax.

  11. Add a label (ideally one that matches the associated issue or issues). Add a project as well to get the PR to automatically have a card generated on the appropriate project page. Only one project should be added, with that project being the one that is associated with the target branch of the PR. Also add a milestone, if applicable. Once finished, create the pull request.

  12. DO NOT ADD REVIEWERS.

  13. Once the PR has moved up the queue, the scheduler will assign reviewers and move the PR to the "Upcoming tags - in review and testing" column. Reviewers should follow these guidelines:

    • Review code, data, and documentation changes, adding comments, and change requests as necessary.
    • If you are designated as the "first reviewer" for a PR by you being the sole reviewer and the review is marked "Draft", when your review is complete and all requested changes have been made, remove the "Draft" designation and assign the other reviewers.
    • Recognize that once a review is requested, these are the next potential tags, so be sure to act quickly on the review and re-review requests.
    • If you do not have time to fill the review request, or do not feel you have the necessary expertise, you may remove yourself from the reviewers list. Note the reason for your decision in the PR conversation section.
    • There needs to be at least two approved reviews for a PR so not everyone can abstain from a requested review.
    • Only "Approve" a PR if the code can go in as-is with no changes. Otherwise please select either "Comment" or "Request changes" when leaving a review.

  14. Make requested changes to your personal branch (following the above instructions for modifying your fork).

IMPORTANT NOTE for git-fleximod: Now is a good time to make sure that any externals which have been updated (as seen with a git status) have been git added to the commit. See Updating directories for git-fleximod for more information.

  1. Notify reviewers that you are ready for a re-review. Click on the two arrows to the right of the reviewers name on the github PR page. Note that while they may receive automated emails from GitHub as you push changes, requesting re-review is a necessary step to indicate you have completed your modifications.

    • If the PR appears to be ready but there are still pending reviews, first notify those reviewers and ask for an expedited review. If the reviewer is not available, the acting scheduler can decide to eliminate that review so the PR can proceed.
    • A reviewer should mark a comment of theirs as resolved (by clicking the 'Resolve conversation' button for that comment), however, no one else should take this action so that the reviewer has a chance to ensure that the issue has been adequately addressed.
    • If multiple people are involved in a conversation, the original poster may explicitly pass off responsibility for the issue to another reviewer (flag them in the conversation).
    • All issues (conversations) should be resolved before a PR is merged.

  2. When all reviewers have approved the PR, you will need to receive notice that your commit is next in line from the scheduler (currently Cheryl Craig, and Jesse Nusbaumer or Courtney Peverley usually fill in when she is out). At this point, you need to:

    • Make sure that your code is current with the head of the branch to which you are committing
    • Rerun bin/git-fleximod update if any External config files have changed
    • Run all the tests with this code base
    • Update the ChangeLog filling in the testing results, today's date and the tag name that you've been given and push it back to your branch

  3. If this is your first time making a tag, please check with the person who gave you your tag number that you have completed all of the steps prior to pushing the "merge" button. At this point, your code may be merged with the appropriate upstream branch (e.g. "cam_development"). When you click the "merge" button, a dialog box will pop up with a title that will like this:

Merge pull request #XXX from <username>/<branch_name>

where #XXX is the PR number, <username> is your Github username, and <branch_name> is the name of the branch that contains the modifications. Please do NOT edit this title, as automated scripts depend on the text within the title in order to function properly. However, you can add anything you want in the comment box below the title.

  1. Create a new tag

  2. Store the baselines on both derecho and izumi (run archive_baseline.sh) Use CESM_TESTDIR to identify the location of your testing directory.

On izumi:

  • env CESM_TESTDIR=/scratch/cluster/YourName/aux_cam_gnu_yyyymmddsssss CAM_FC=GNU ./archive_baseline.sh cam6_3_XXX

  • env CESM_TESTDIR=/scratch/cluster/YourName/aux_cam_nag_yyyymmddsssss CAM_FC=NAG ./archive_baseline.sh cam6_3_XXX

On derecho:

  • env CESM_TESTDIR=/glade/derecho/scratch/YourName/aux_cam_yyyymmddsssss CAM_FC=INTEL ./archive_baseline.sh cam6_3_XXX

  • env CESM_TESTDIR=/glade/derecho/scratch/YourName/aux_cam_yyyymmddsssss CAM_FC=NVHPC ./archive_baseline.sh cam6_3_XXX

  1. Add the tag to the appropriate location on the CESM "plans" page (https://csegweb.cgd.ucar.edu/testdb/cgi-bin/tagList.cgi), if necessary.
  • Determine where the tag should be placed (which upcoming tag it should go in). If you have questions on the proper tag, contact Chris Fischer ([email protected]) or Cheryl Craig ([email protected]) and they can advise you with the appropriate tag.
  • Click on the appropriate tag
  • Scroll down to CAM and hit the + sign which opens up a window to fill out
  • revision is the tag name you just made (note it will automatically change to a fullpath name when it is entered into the table)
  • Indicate if it is bit-for-bit, some or none
  • Enter in the PR title and if the changes are "some" indicate which tests are different
  • Indicate if this tag requires any other external components to be updated at the same time (a coordinated CESM tag) and enter that information in the notes as well

Run CAM tests

Make sure that you have merged your branch up to the HEAD of the cam_development or release branch prior to running tests. Rerun bin/git-fleximod update if any Externals config files have changed. If your branch is behind, merge in your testing sandbox but do not push the merge commit back to your GitHub remote.

  • On Izumi: Run <root>/test/system/test_driver.sh with CAM_FC=gnu and CAM_FC=nag and BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/XXXX (where XXXX is the name of the previous baseline)

    • env BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/XXXX_nag CAM_FC=NAG ./test_driver.sh -f
    • env BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/XXXX_gnu CAM_FC=GNU ./test_driver.sh -f

  • On Derecho: Run <root>/test/system/test_driver.sh with CAM_FC=intel and CAM_FC=nvhpc and BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/XXXX (where XXXX is the name of the previous baseline)

    • env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/XXXX_intel CAM_FC=intel ./test_driver.sh -f
    • env BL_TESTDIR=/glade/campaign/cesm/community/amwg/cam_baselines/XXXX_nvhpc CAM_FC=nvhpc ./test_driver.sh -f

  • Review tests (note the ONLY tests which are allowed to fail are ones expected due to changes in the commit and every one will need to be documented in the ChangeLog)

    • For CESM tests, look in the aux_cam_zzz_yyyymmddsssss.log file for the --test-root location. cd to the test-root directory and run the cs.status.aux_cam_zzz_yyyymmddsssss command. It is useful to run grep to find the lines of interest. Note that currently the MEMCOMP test is broken and TPUTCOMP gives inconsistent information. We are currently ignoring these tests. For example to remove all the lines with PASS and the bad tests (to more easily see the anomalies) run ./cs.status.aux_cam_zzz_yyyymmddsssss | grep -v PASS | grep -v TPUTCOMP | grep -v MEMCOMP.
    • Be sure to look in aux_cam_nag_yyyymmmddsssss.log file for the output of the TR8 and TGIT tests. Fix any errors listed there.

  • If most tests pass, but a few tests had errors due to machine glitches, then the following procedure can be followed. Do not use this process if you needed to make any code changes. In that case, you must re-run the entire test suite

    • Temporarily edit the <root>/cime_config/testdefs/testlist_cam.xml file to contain tests with a new name (in this example it will be aux_run_failing) and add tests for all the failed tests. This changed file should only be used for your testing and not committed or pushed to the PR branch.
    • use the --cesm flag for test_driver.sh. For example: env CESM_TESTDIR=/full/path/from/test-root CAM_FC=NAG BL_TESTDIR=/fs/cgd/csm/models/atm/cam/pretag_bl/XXXX_nag ./test_driver.sh -f --cesm aux_run_failing
    • when these tests all pass, you will need to run two archive baselines for this machine/compiler
      • archive the main aux_cam test runs to cam6_3_XXX
      • archive the aux_run_failing test runs to a temporary unique name (in this example temp_failed_reruns)
      • go into /fs/cgd/csm/models/atm/cam/pretag_bl/cam6_3_XXX_compiler and remove the files in the directories of the tests which failed
      • copy the contents of each subdirectory in temp_failed_reruns into the appropriate directory in cam6_3_XXX_compiler
      • when done, remove the temporary directory (temp_failed_reruns)

  • Fix any errors and/or document expected fails

  • Make a science run, if is is required (e.g., for answer changing updates or for new tests)

Make a CAM tag

  • Make sure all CAM tests have been run on the HEAD of the destination branch (e.g., cam_development or release branch).
    • To clarify, this means that you either merged the HEAD of the destination branch into your branch or rebased your branch on top of the HEAD of the destination branch before your last round of tests.
  • Update ChangeLog to reflect all changes since previous tag on branch
  • Commit to branch and push to proper remote
  • Open a pull request from your branch to the correct branch on ESCOMP.
  • After pull request has been merged (see below), make an annotated tag with the ChangeLog summary as the tag commit message
    1. Clone https://github.com/ESCOMP/CAM or update an existing clone with git fetch <escomp_remote> <branch_name>.
    2. Find the PR merge commit (<merge_commit>) for the tag:
      • In the ESCOMP/CAM clone, find the merge commit with git log --oneline -1 <escomp_remote>/<branch_name> where the merge commit is the first field in the output.
    3. Tag the <merge_commit> with git tag -a <tag> <merge_commit> -m '<commit_message'> where<commit_message> is the same as the message for the <merge_commit>
  • Push the tag to the proper remote git push <escomp remote> <tag>

ADDITIONAL SUPPORTING INFORMATION

Handle a pull request to cam_development

  • Make sure the "Create a merge commit" option is selected in the dropdown to the right of the green 'Merge pull request' button.
  • After clicking the button, copy and paste the entire text of the PR description (first item in the PR 'conversation' where the ChangeLog template is filled out).
  • Click "Confirm merge".
  • Update your cam_development branch in a clone that can run a test.
  • Check the log to see that your commit is there.
  • Run at least one test to make sure the code works.
  • Delete the branch which was just merged.

Make a code change in release branch and cam_development

Update development branch from cam_development or release branch

Before running tests or merging in a PR, you should update your code to the head of the target branch (e.g., cam_development). Before opening the PR, the recommended procedure is to rebase your branch onto the target branch. For example, if the remote is named ESCOMP and the target branch is cam_development, make sure your branch is clean and run:

git rebase ESCOMP/cam_development

After this, your branch should look as if it was just created on top of the HEAD of cam_development. Push this branch back to your fork using the --force option.

For more information on using git rebase, see the description from the Pro Git book or the manual page (which includes git rebase --interactive).

However, if, after opening a PR, another tag is made before you, you should merge the changes into your branch:

git merge ESCOMP/cam_development

For more on git merge, see the description from the Pro Git book or the manual page.

Updating directories for git-fleximod

When you update externals, there is an additional step you need to make now that we are using git-fleximod. In addition to committing the .gitmodules file, you also need to commit the directories which are used for the modified externals (after you run git-fleximod). Without this step, subsequent "git status" messages will say that the directory has been modified. This doesn't cause an error, but is just a nuisance message (which can lead to confusion).

You will see the directories that need to be committed after you run "bin/git-fleximod update" and do a "git status"

Example:

If you updated .gitmodules with a new tag for src/atmos_phys, git status will report

modified:   src/atmos_phys (new commits)

Then you will need to:

git add src/atmos_phys

You may periodically see directories for external components which are not part of CAM (i.e. CTSM or CICE) with this same message and these will reside in the components directory. Do NOT add these directories, only commit ones which start with the src directory.

Create a new release

CAM wiki

CAM Documentation

CAM Model Development

Clone this wiki locally