Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Build conda and environments in SRW. #938

Merged
merged 50 commits into from
Nov 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
5a38a5d
WIP.
christinaholtNOAA Oct 6, 2023
2a5b05a
Make the fundamental tests run.
christinaholtNOAA Oct 7, 2023
aeccb68
Modify the rest of the modulefiles accordingly.
christinaholtNOAA Oct 7, 2023
4611e70
Modifying the GHA workflows to use the environment
christinaholtNOAA Oct 7, 2023
c8f4cf9
Merge remote-tracking branch 'origin' into build_conda
christinaholtNOAA Oct 7, 2023
4bdf0bc
Attempt to fix mamba
christinaholtNOAA Oct 7, 2023
c9735bc
Fix autosyntax error.
christinaholtNOAA Oct 7, 2023
64c5369
Cleanup old flags.
christinaholtNOAA Oct 7, 2023
255765c
Try to install env with mamba install.
christinaholtNOAA Oct 7, 2023
e6bb3a3
Try better default shell for activating mamba
christinaholtNOAA Oct 7, 2023
4421031
Perhaps a typo in the micromamba docs?
christinaholtNOAA Oct 7, 2023
9ee47c3
Don't separate jobs
christinaholtNOAA Oct 7, 2023
0f8b58a
Reorder and cleanup.
christinaholtNOAA Oct 7, 2023
66202d7
Update docs.
christinaholtNOAA Oct 7, 2023
5c644d8
Try different syntax
christinaholtNOAA Oct 7, 2023
d3329d4
Check if this limits this to only mac.
christinaholtNOAA Oct 7, 2023
a0e047e
Revert "Check if this limits this to only mac."
christinaholtNOAA Oct 7, 2023
17c5af2
Fixes for building on MacOs.
christinaholtNOAA Oct 10, 2023
16796b6
Merge remote-tracking branch 'ufs/develop' into build_conda
christinaholtNOAA Oct 10, 2023
2fcc2b2
Configurable conda location.
christinaholtNOAA Oct 10, 2023
d11254c
Make it MacOS compatible.
christinaholtNOAA Oct 10, 2023
c7b2cc8
format note block
gspetro-NOAA Oct 12, 2023
4a19ec5
Rm dead link to MacMorePackages
gspetro-NOAA Oct 12, 2023
1466e14
Update docs/UsersGuide/source/BuildingRunningTesting/RunSRW.rst
christinaholtNOAA Oct 17, 2023
1caae96
Updating docs with Gillian's feedback.
christinaholtNOAA Oct 17, 2023
f370f2c
Merge branch 'build_conda' of https://github.com/christinaholtNOAA/uf…
christinaholtNOAA Oct 17, 2023
a2410ef
Update quickstart, too
christinaholtNOAA Oct 17, 2023
c2b6d2e
Reverting changes for containers.
christinaholtNOAA Oct 18, 2023
3fd7f2f
Reverting changes to VX chapter.
christinaholtNOAA Oct 18, 2023
bd8009f
Update devbuild.sh
christinaholtNOAA Oct 18, 2023
db0de87
Don't overwrite defaults.
christinaholtNOAA Oct 18, 2023
796512f
Merge branch 'build_conda' of https://github.com/christinaholtNOAA/uf…
christinaholtNOAA Oct 18, 2023
0e03b39
Update devbuild.sh
christinaholtNOAA Oct 18, 2023
ec19437
Addressing some of Mike's review comments.
christinaholtNOAA Oct 18, 2023
a5fc945
Merge branch 'build_conda' of https://github.com/christinaholtNOAA/uf…
christinaholtNOAA Oct 18, 2023
09c3917
Fix issue with URL.
christinaholtNOAA Oct 19, 2023
79d9abc
Turn conda on by default, off for AQM.
christinaholtNOAA Oct 20, 2023
e666780
Remove conda target from test build script.
christinaholtNOAA Oct 20, 2023
a34c0e4
Addressing Gillian's comments.
christinaholtNOAA Oct 23, 2023
579c2e3
Update tests/README.md
christinaholtNOAA Oct 30, 2023
e4e9a27
Update docs/UsersGuide/source/BuildingRunningTesting/BuildSRW.rst
christinaholtNOAA Oct 30, 2023
47860e5
Adding AQM to the supported environments
christinaholtNOAA Nov 20, 2023
59039e8
Adding aqm environment file.
christinaholtNOAA Nov 20, 2023
a358c52
Fix a typo for creating environment.
christinaholtNOAA Nov 20, 2023
65b05b7
Merge remote-tracking branch 'ufs/develop' into build_conda
christinaholtNOAA Nov 20, 2023
18389b9
Merge branch 'build_conda' of https://github.com/christinaholtNOAA/uf…
christinaholtNOAA Nov 21, 2023
fff3a25
Fixing python environments for AQM.
christinaholtNOAA Nov 21, 2023
387373e
Address review comments.
christinaholtNOAA Nov 27, 2023
1e3a8f9
Changes for Jenkins pipeline failures.
christinaholtNOAA Nov 27, 2023
e9b6318
Updating docs.
christinaholtNOAA Nov 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .cicd/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -137,16 +137,6 @@ pipeline {
}
}

// Run the unittest functional tests that require an HPC platform
stage('Functional UnitTests') {
steps {
dir ("${env.SRW_PLATFORM}") {
echo "Running unittest on retrieve_data.py"
sh 'bash --login "${WORKSPACE}/${SRW_PLATFORM}/.cicd/scripts/srw_unittest.sh"'
}
}
}

// Run the unified build script; if successful create a tarball of the build and upload to S3
stage('Build') {
steps {
Expand All @@ -164,6 +154,16 @@ pipeline {
}
}

// Run the unittest functional tests that require an HPC platform
stage('Functional UnitTests') {
steps {
dir ("${env.SRW_PLATFORM}") {
echo "Running unittest on retrieve_data.py"
sh 'bash --login "${WORKSPACE}/${SRW_PLATFORM}/.cicd/scripts/srw_unittest.sh"'
}
}
}

// Try a few Workflow Task scripts to make sure E2E tests can be launched in a follow-on 'Test' stage
stage('Functional WorkflowTaskTests') {
steps {
Expand Down
2 changes: 1 addition & 1 deletion .cicd/scripts/srw_ftest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ module load build_${platform,,}_${SRW_COMPILER}
module load wflow_${platform,,}

[[ ${FORGIVE_CONDA} == true ]] && set +e +u # Some platforms have incomplete python3 or conda support, but wouldn't necessarily block workflow tests
conda activate workflow_tools
conda activate srw_app
set -e -u

export PYTHONPATH=${workspace}/ush/python_utils/workflow-tools:${workspace}/ush/python_utils/workflow-tools/src
Expand Down
39 changes: 0 additions & 39 deletions .github/workflows/python_func_tests.yaml

This file was deleted.

40 changes: 0 additions & 40 deletions .github/workflows/python_linter.yaml

This file was deleted.

58 changes: 58 additions & 0 deletions .github/workflows/python_tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: Python Tests
on:
push:
branches:
- develop
- 'release/*'
pull_request:
branches:
- develop
- 'release/*'
workflow_dispatch:

defaults:
run:
shell: bash -leo pipefail {0}
gspetro-NOAA marked this conversation as resolved.
Show resolved Hide resolved

jobs:
srw_tests:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v2

- name: Install Micromamba and srw_app environment
uses: mamba-org/setup-micromamba@v1
with:
environment-file: environment.yml
environment-name: srw_app
init-shell: bash
cache-downloads: true
cache-environment: true

- name: Checkout externals
run: |
./manage_externals/checkout_externals ufs-weather-model workflow-tools

- name: Lint the python code
run: |
micromamba activate srw_app
export PYTHONPATH=$(pwd)/ush:$(pwd)/ush/python_utils/workflow-tools:$(pwd)/ush/python_utils/workflow-tools/src
gspetro-NOAA marked this conversation as resolved.
Show resolved Hide resolved
pylint --ignore-imports=yes tests/test_python/
pylint ush/create_*.py
pylint ush/generate_FV3LAM_wflow.py

- name: Run python unittests
run: |
# exclude test_retrieve_data that is tested in functional test
micromamba activate srw_app
export UNIT_TEST=True
export PYTHONPATH=$(pwd)/ush:$(pwd)/ush/python_utils/workflow-tools:$(pwd)/ush/python_utils/workflow-tools/src
python -m unittest -b tests/test_python/*.py

- name: Run python functional tests
run: |
micromamba activate srw_app
export CI=true
export PYTHONPATH=${PWD}/ush
python3 -m unittest -b tests/test_python/test_retrieve_data.py
gspetro-NOAA marked this conversation as resolved.
Show resolved Hide resolved
44 changes: 0 additions & 44 deletions .github/workflows/python_unittests.yaml

This file was deleted.

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ ush/config.yaml
ush/python_utils/__pycache__/
ush/python_utils/workflow-tools/
ush/*.swp
conda_loc

*.swp
__pycache__
13 changes: 13 additions & 0 deletions aqm_environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: srw_aqm
channels:
- conda-forge
- ufs-community
dependencies:
- esmpy
- netcdf4=1.6.*
- numpy=1.23.*
- pylint=2.17*
- pytest=7.2*
- scipy=1.10.*
- uwtools=1.0.0
- xarray=2022.11.*
75 changes: 64 additions & 11 deletions devbuild.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ OPTIONS
installation prefix
--bin-dir=BIN_DIR
installation binary directory name ("exec" by default; any name is available)
--conda-dir=CONDA_DIR
installation location for miniconda (SRW clone conda subdirectory by default)
--build-type=BUILD_TYPE
build type; defaults to RELEASE
(e.g. DEBUG | RELEASE | RELWITHDEBINFO)
Expand All @@ -52,9 +54,11 @@ OPTIONS
build with verbose output

TARGETS
default = builds the default list of apps (also not passing any target does the same)
all = builds all apps
Or any combinations of (ufs, ufs_utils, upp)
default = builds the default list of components for the specified application
(also not passing any target does the same)
all = builds all standard components for ATM
conda_only = installs miniconda, but no other
Or any combinations of (ufs, ufs_utils, upp, nexus, aqm_utils)

NOTE: See User's Guide for detailed build instructions

Expand Down Expand Up @@ -99,11 +103,8 @@ usage_error () {

# default settings
LCL_PID=$$
SRW_DIR=$(cd "$(dirname "$(readlink -f -n "${BASH_SOURCE[0]}" )" )" && pwd -P)
MACHINE_SETUP=${SRW_DIR}/src/UFS_UTILS/sorc/machine-setup.sh
BUILD_DIR="${SRW_DIR}/build"
INSTALL_DIR=${SRW_DIR}
BIN_DIR="exec"
CONDA_BUILD_DIR="conda"
COMPILER=""
APPLICATION=""
CCPP_SUITES=""
Expand All @@ -117,6 +118,7 @@ VERBOSE=false

# Turn off all apps to build and choose default later
DEFAULT_BUILD=true
BUILD_CONDA="on"
BUILD_UFS="off"
BUILD_UFS_UTILS="off"
BUILD_UPP="off"
Expand Down Expand Up @@ -164,6 +166,8 @@ while :; do
--install-dir|--install-dir=) usage_error "$1 requires argument." ;;
--bin-dir=?*) BIN_DIR=${1#*=} ;;
--bin-dir|--bin-dir=) usage_error "$1 requires argument." ;;
--conda-dir=?*) CONDA_BUILD_DIR=${1#*=} ;;
--conda-dir|--conda-dir=) usage_error "$1 requires argument." ;;
--build-type=?*) BUILD_TYPE=${1#*=} ;;
--build-type|--build-type=) usage_error "$1 requires argument." ;;
--build-jobs=?*) BUILD_JOBS=$((${1#*=})) ;;
Expand All @@ -175,6 +179,7 @@ while :; do
default) ;;
all) DEFAULT_BUILD=false; BUILD_UFS="on";
BUILD_UFS_UTILS="on"; BUILD_UPP="on";;
conda_only) DEFAULT_BUILD=false;;
ufs) DEFAULT_BUILD=false; BUILD_UFS="on" ;;
ufs_utils) DEFAULT_BUILD=false; BUILD_UFS_UTILS="on" ;;
upp) DEFAULT_BUILD=false; BUILD_UPP="on" ;;
Expand All @@ -188,10 +193,9 @@ while :; do
done

# Ensure uppercase / lowercase ============================================
APPLICATION="${APPLICATION^^}"
PLATFORM="${PLATFORM,,}"
COMPILER="${COMPILER,,}"
EXTERNALS="${EXTERNALS^^}"
APPLICATION=$(echo ${APPLICATION} | tr '[a-z]' '[A-Z]')
PLATFORM=$(echo ${PLATFORM} | tr '[A-Z]' '[a-z]')
COMPILER=$(echo ${COMPILER} | tr '[A-Z]' '[a-z]')

# check if PLATFORM is set
if [ -z $PLATFORM ] ; then
Expand All @@ -203,6 +207,55 @@ fi
MACHINE="${PLATFORM}"
printf "PLATFORM(MACHINE)=${PLATFORM}\n" >&2


# Conda is not used on WCOSS2
if [ "${PLATFORM}" = "wcoss2" ]; then
BUILD_CONDA="off"
fi

# build conda and conda environments, if requested.
if [ "${BUILD_CONDA}" = "on" ] ; then
if [ ! -d "${CONDA_BUILD_DIR}" ] ; then
os=$(uname)
test $os == Darwin && os=MacOSX
hardware=$(uname -m)
installer=Miniforge3-${os}-${hardware}.sh
curl -L -O "https://github.com/conda-forge/miniforge/releases/download/23.3.1-1/${installer}"
bash ./${installer} -bfp "${CONDA_BUILD_DIR}"
rm ${installer}
fi

source ${CONDA_BUILD_DIR}/etc/profile.d/conda.sh
# Put some additional packages in the base environment on MacOS systems
if [ "${os}" == "MacOSX" ] ; then
mamba install -y bash coreutils sed
gspetro-NOAA marked this conversation as resolved.
Show resolved Hide resolved
fi
conda activate
if ! conda env list | grep -q "^srw_app\s" ; then
mamba env create -n srw_app --file environment.yml
fi
if ! conda env list | grep -q "^srw_graphics\s" ; then
mamba env create -n srw_graphics --file graphics_environment.yml
fi
if [ "${APPLICATION}" = "ATMAQ" ]; then
if ! conda env list | grep -q "^srw_aqm\s" ; then
mamba env create -n srw_aqm --file aqm_environment.yml
fi
fi

else
source ${CONDA_BUILD_DIR}/etc/profile.d/conda.sh
conda activate
gspetro-NOAA marked this conversation as resolved.
Show resolved Hide resolved
MichaelLueken marked this conversation as resolved.
Show resolved Hide resolved
fi

# Conda environment should have linux utilities to perform these tasks on macos.
SRW_DIR=$(cd "$(dirname "$(readlink -f -n "${BASH_SOURCE[0]}" )" )" && pwd -P)
MACHINE_SETUP=${SRW_DIR}/src/UFS_UTILS/sorc/machine-setup.sh
BUILD_DIR="${BUILD_DIR:-${SRW_DIR}/build}"
INSTALL_DIR=${INSTALL_DIR:-$SRW_DIR}
CONDA_BUILD_DIR="$(readlink -f "${CONDA_BUILD_DIR}")"
echo ${CONDA_BUILD_DIR} > ${SRW_DIR}/conda_loc
MichaelLueken marked this conversation as resolved.
Show resolved Hide resolved

# choose default apps to build
if [ "${DEFAULT_BUILD}" = true ]; then
BUILD_UFS="on"
Expand Down
Loading
Loading