-
Notifications
You must be signed in to change notification settings - Fork 0
CableUserGuide_AccessUm
In this section we discuss building and then running CABLE online as the land-surface model (LSM) in the UM. Specifically we discuss ACCESS model versions of UM+CABLE. Here we assume that the host and target machines areaccessdev.nci.org.au (accessdev) and raijin.nci.org.au (raijin), respectively.
There are currently four variants of the UM+CABLE version of the ACCESS model. These are listed in the table below along with paths to UM source code for each model, and for CABLE source code the applicable revision in the trunk.
The paths to the UM source code are relative to the root of the repository. This is
"https://access-svn.nci.org.au/svn/um".
The CABLE trunk can be found at
"https://trac.nci.org.au/svn/cable/trunk".
Distinct from CABLE-2.0 versions of the UM+CABLE, in the ACCESS1.3 model CABLE (approx. version 1.8) is integrated into the UM source code.
ACCESS1.3 is included in the table for completeness but is not discussed further here.
Similarly, ACCESS2.0 is included in the table although is still under development and is not discussed further.
Finally, we also include in the table example UMUI jobs for each model.
#!html
<table cellpadding=5 cellspacing=0 style='background-color:#f8f8f8; border-top: 1px solid black; border-bottom: 1px solid black;'>
<tbody>
<tr><th>Model</th> <th>UM</th> <th>CABLE</th> <th>UMUI</th></tr>
<tr><td>ACCESS1.3</td> <td>^branches/pkg/Rel/ACCESS1.3</td> <td>NA</td> <td>vagzB, vagzA</td></tr>
<tr><td>ACCESS1.3b</td> <td>^branches/pkg/Rel/ACCESS-1.3_CABLE-2.0_replacement</td> <td>797</td> <td>vaheA</td></tr>
<tr><td>ACCESS1.4</td> <td>^branches/pkg/Rel/ACCESS-1.4</td> <td>8122</td> <td>vaheB</td></tr>
<tr><td>ACCESS2.0</td> <td>TBA</td> <td>TBA</td> <td>TBA</td></tr>
</tbody>
</table>
In this section we discuss building CABLE as a library (Section 4.1), linking it into the UM (Section 4.2) and then running CABLE online (Section 4.3) as the LSM in the UM. This discussion pplies to versions ACCESS1.3b and ACCESS1.4.
Having downloaded CABLE, go into the UM directory (cd UM/). Other than source code, this directory contains a Makefile (Makefile_CABLE-UM), and an executable build script (build.ksh).
The first step in the CABLE build process is to determine what Fortran compiler you use and what compile flags to prepend to the build, and the location of netcdf files to be linked. This is automatic for raijin which is where most/all ACCESS+CABLE jobs are currently being run. Otherwise build.ksh will prompt you for these details, and if you so choose, will save these details to the script. If this is a machine that you believe CABLE should support for all users then please email us at [mailto:[email protected] [email protected]] and we will make the changes permanent. Remember that you will need to checkout and build the CABLE code on the machine that you intend to run the model on, even if you usually run the UMUI on another machine.
The second step for build.ksh is to gather the files appropriate for building. The script creates a hidden directory called .tmp if there isn’t one already, and copies everything into .tmp. You should never go into the .tmp directory. One of the features of the build process is that only source files which are modified are re-built, and their dependents. This is possible because the .tmp directory is overwritten by build.ksh, preserving timestamps of the source files from their original location. Moreover, if you change the files in .tmp directly, those changes will not be picked up by the next build and worse still, they will be overwritten and lost.
For working copies of CABLE that are under version control, the build script passively queries what revision number you have and writes this to a hidden file in your home directory. This revision number is then included into the output log. This is far from foolproof, and should only be used as a guide. Some obvious flaws in this methodology are that the timing of multiple builds and subsequent runs may render the revision data sent to the output file incorrect. The revision number written to the output file reflects the last time you svn updated your working copy, which could be misleading. The build script also queries the status of your working copy. The output log then contains the result of this query. If there are any local modifications, then the first ten modified files are displayed. At least then you will know that the code being built is not only the revision reported. We have decided this method to be preferable to the built in svn substitution method.
Next build.ksh submits, Makefile_CABLE-UM, which basically submits the compile instructions. You will probably never have to go into Makefile_CABLE-UM, unless you add a file to CABLE. Finally , the build script creates a library, libcable.a, from the CABLE object files and copies this library to the CABLE- AUX/UM/ directory where it will be picked up by the UM build. Any existing libcable.a here will be copied to a dated backup. Upon execution of the UM, libcable.a will be copied to the UM run directory. A dated backup will be created herein as well, should a previous libcable.a be present. This is also not fool proof but does offer some recourse should it ever be required.
In the last section we discussed building the CABLE library (libcable.a). Given that libcable.a is in place, building the UM with CABLE basically involves using the UMUI job described in Section 4.2.2.At present we support UM version 7.3.
As discussed in Section 6.3, there are four calls to CABLE from the UM. In addition to these four calls, CABLE affects approximately 40 files in the UM source code. These modifications are generally either to allow for CABLE to operate with different vegetation types and number of soil levels to the MOSES land surface scheme, to pass CABLE variables through the UM’s input/output system (STASH), or to turn off calls to MOSES routines. This UM code is equivalent to that run in ACCESS1.3 CMIP5 simulations Bi et al., AMOJ, 2013, except for some changes to how CABLE is interfaced with the UM for CABLE-2.0 instead of CABLE-1.8. Note that these branches also contain the extended tracer code capabilities described in Corbin and Law (2011).The UM source code appropriate for ACCESS models can be found in the UM repository. Specific paths are detailed in the above table.
For users intending to build ACCESS1.3b or ACCESS1.4, using the UMUI (Section 4.2.1), the UM branches are linked in via the specified FCM configuration in the example jobs. In short, there is no need to ever come into contact with this UM code.
The UMUI is designed to run on a machine remote to where the model itself builds and runs. We are linking a CABLE library into the UM build, and so that library should preferably be built on the same machine as the UM. The CABLE library is otherwise foreign to the UM, and so the UM build process needs to be specially configured to look for this library in the proper location. Unlike the GCOM libraries etc, the CABLE library is potentially subject to frequent change, by different users. Therefore, we can’t enforce a static location that is general to all users. However, each user can uniquely define a $CABLE_AUX environment variable that can be used to set the path to their library, or indeed their installation of CABLE-AUX (Section 2.2) generally. Note, whilst standardising an ACCESS1.3b UMUI job for raijin/accesdev, we noted several instances where dereferencing $CABLE_AUX was problematic. However, $HOME can always be properly dereferenced. Therefore the standardised UMUI jobs for ACCESS1.3b and ACCESS1.4 assume your installation of CABLE-AUX is in your home directory. (i.e. $CABLE_AUX = $HOME)
The CABLE library ($CABLE_AUX/UM/libcable.a) can then be linked into the UM build. Please note that we use the $CABLE_AUX environment variable to derive necessary paths within some scripts as well.
You will need to set the $CABLE_AUX environment variable for the shell you use on both the machine that you are running the UMUI (e.g. accessdev) and also on the machine that you are building and running ACCESS (e.g. raijin). Should you use csh, for example, then include in your .citric
setenv CABLE_AUX PathToYourCABLE-AUXinstallation e.g. $HOME/CABLE-AUX
Note: It is necessary to set the environment variable prior to launching UMUI.
The override file (raijin_libcable.ovr), configures the UM+CABLE build. The hand-edit file (cable_handedits.ed),
configures both build and run time behaviours for UM+CABLE. By default, the standardised UMUI job points to global override and handedit files.
It is unlikely you will want to change these files. However, should
the need arise, or you are submitting UM jobs from another machine, these files are available in the
CABLE-AUX/UMUI subdirectory. Note: You may need to reset some paths in the UMUI as well.
Irrespective of whether you are using a default job or not, you will need to set the $CABLE_AUX environment variable in ksh (in order to use the ‘top script’ described in Section 4.3), on the machine that you are building and running ACCESS . Include in your .kshrc (or create this file if you do not have one)
export CABLE_AUX='PathToYourCABLE-AUXinstallation
Finally, once all hand-edits and over-rides are specified as stated above, $CABLE_AUX and .kshrc have been set, then you can simply process and submit your build job as you normally would.
Example, standard UMUI jobs are provided for ACCESS1.3b and ACCESS1.4 on accessdev. The experiment IDs are vaheA, and vaheB, respectively. This job is equivalent to merged CMIP5 build/run jobs, with modifications to accommodate CABLE-2.x. We have intentionally supplied a single job to service both build and run functions. The default setting is to compile and then run.
vaheA and vaheB, are intended for users to build their own code, either UM or CABLE, into the final executable. Should the user wish to use their own UM code, it is advised they first branch from corresponding paths identified in the above table, and make their modifications in the new branch. Users will then need to edit the FCM configuration section in the UMUI to include a local working copy and/or their own branch in the UM repository. The UM build process incorporates CABLE through $CABLE_AUX/CABLE-AUX/UM/libcable.a. Pre-compiling libcable.a is discussed in preceding sections.
Running a UM job which uses CABLE is effectively the same as running any other UM job. We have modified the UM’s input/output system (STASH) and provided the proper initialisation datasets so that CABLE can operate with its familiar resolution of 17 tiles (11 PFTs, 4 Non-veg tiles, 2 spare tiles), 6 soil layers and 3 snow layers. The included hand-edit file (cable_handedits.ed) updates some runtime variables to this end.
One of the very first things CABLE does is to look for its namelist file (cable.nml). This namelist configures many aspects of CABLE and is discussed in Section 5.1.1.Note that the offline version of CABLE uses many more namelist options than the UM code. In the UM, a top script, cable_start.sh copies cable.nml in $CABLE_AUX/CABLE-AUX/UM into the UM run directory on start up, unless there is one there already. This allows users to be running multiple UM experiments concurrently with different CABLE configurations. Aside from model options, cable.nml also specifies the path to essential input data such as PFT dependent vegetation parameters and soil type dependent soil parameters.
CABLE output from ACCESS simulations is handled through the UM Storage Handling and Diagnostic System (STASH). A large range of variables can by output at user-defined spatial and temporal frequencies. These are set through the UMUI under Atmosphere/STASH/STASH. Specification of Diagnostic requirement. Example output specifications can be seen in the example AMIP job, including the output of global time-averaged fields and hourly timeseries from a selection of grid-cells (e.g. flux tower locations). Some CABLE output uses STASH variables that had already been created for output from the MOSES land surface scheme (usually from Section 0 or Section 3). Other fields, such as tiled soil variables, required new STASH variables. Note that tiled soil variables require separate STASH variables for each soil level. A table of CABLE output and its associated STASH variable is given in Appendix E.
Output from the STASH system is in a UM binary format. Conversion to netcdf is possible through xconv or a convsh script (although timeseries output needs to be handled differently). The ACCESS land surface modeling team has developed some tools for post-processing and plotting ACCESS output. Further information is available from [mailto:[email protected] [email protected]].