Add flexible controls of GPU configuration

[sjsprecious]

This PR updates the configurations for the GPU offload based on the PRs in CMEPS (ESCOMP/CMEPS#363) and ccs_config_cesm (ESMCI/ccs_config_cesm#97).

An example to build a GPU case on Derecho with these new GPU options by using create_newcase looks:
./create_newcase --case /path_to_case_dir --mach derecho --compiler nvhpc --mpilib mpich --compset F2000dev --res f19_f19_mg17 --queue main --ngpus-per-node 4 --gpu-type a100 --gpu-offload openacc

An example to build a GPU case on Derecho with these new GPU options by using create_test looks:
./create_test ERP_Ln9_G4-a100-openacc.f19_f19_mg17.F2000dev.gust_nvhpc.cam-outfrq9s --test-root /path_to_case_dir --output-root /path_to_output -q main

[sjsprecious]

@jedwards4b will issue a separate PR to introduce these changes.

[rljacob]

I believe this works independently of the E3SM approach to define gpu-specific compilers? And I think we would handle a machine like derecho with separate machine entries for each node type.

[jedwards4b]

that's correct - we are departing from that approach.

[jedwards4b]

@sjsprecious It looks like you may need to add something to make sure this only happens when driver=nuopc

[sjsprecious]

Thanks @jedwards4b . Do you mean that we should only use these new GPU options for the NUOPC driver? Why can't we use it for MCT?

[jedwards4b]

If you want use them for MCT then you will need to make changes in
components/cpl7/driver/cime_config/config_component.xml
similar to what you did in cmeps.

[sjsprecious]

Thanks @jedwards4b for your explanation.

So what is the best way you think to handle the MCT case? Currently generating a case with MCT will fail due to the missing new XML variables as expected. Would you suggest bypassing those errors and continuing to build anyway (probably ignoring the given GPU options and always building a CPU case)?

[jedwards4b]

probably the easiest is to just add them to the mct config_component.xml file.

[sjsprecious]

Thanks @jedwards4b . I just opened a PR to introduce these new XML variables in CPL7 (ESCOMP/CESM_CPL7andDataComps#25). Could you please review and merge it if the PR looks good to you?

[jedwards4b]

@sjsprecious The workflow test for e3sm, mct above needs to pass. Remember that these variables are not defined for e3sm.

[sjsprecious]

Thanks @jedwards4b . Since I am not familiar with the E3SM build process, could you please let me know where I can add those new XML variables for the E3SM build workflow?