Getting Started

Downloading the code

First, make sure that git is installed on your machine.

Then download the ERF repository by typing:

git clone https://github.com/erf-model/ERF.git

Or, to automatically include the necessary submodules when downloading ERF, type:

git clone --recursive https://github.com/erf-model/ERF.git

Git Submodules

When using the submodule to build, it is ideal to properly update and match what is in the repository. Depending on Git version, different commands and options to ensure these match. An example workflow is to run git pull to get the latest commit on your current branch, and then run git submodule update to explicitly update the submodule. This should work for all versions of git which support submodules.

The following example demonstrates a shorter form that combines both commands and requires Git 2.14 or newer:

# Replaces your git pull to use both the updated code and the updated submodule
git pull --recurse-submodules

The following example demonstrates setting defaults in the config file for the repository and requires Git 2.15 or newer:

# Set this once
git config submodule.recurse true
# When configured as true, this will use both the updated code and the updated submodule
git pull
# This option will override any configuration and not update the submodule
git pull --no-recurse-submodules

These example also apply to git checkout. For more details see Git Tools Submodules: https://git-scm.com/book/en/v2/Git-Tools-Submodules

Building

The ERF code is dependent on AMReX, and uses the radiation model (RTE-RRTMGP) which is based on YAKL C++ implementation for heterogeneous computing infrastructure (which are all available as submodules in the ERF repo). ERF can be built using either GNU Make or CMake, however, if radiation model is activated, only CMake build system is supported.

Minimum Requirements

ERF requires a C++ compiler that supports the C++17 standard and a C compiler that supports the C99 standard. Building with GPU support may be done with CUDA, HIP, or SYCL. For CUDA, ERF requires versions >= 11.0. For HIP and SYCL, only the latest compilers are supported. Prerequisites for building with GNU Make include Python (>= 2.7, including 3) and standard tools available in any Unix-like environments (e.g., Perl and sed). For building with CMake, the minimal requirement is version 3.18.

Note

While ERF is designed to work with SYCL, we do not make any guarantees that it will build and run on your Intel platform.

Note

ERF was successfully compiled with the Intel compiler suite (e.g., icx version 2024.1.0). However, for older versions, it may be necessary to use reduced compiler optimization (-O1) to avoid an internal compiler error. For example, ERF was successfully compiled with icpc version 19.1.2.254, with -O2 (CMAKE_BUILD_TYPE = RelWithDebInfo) but TimeIntegration/ERF_advance_dycore.cpp had to be manually compiled with -O1. Your mileage may vary.

Paradigm

ERF uses the paradigm that different executables are built in different subdirectories within the Exec directory. When using gmake (see below), the user/developer should build in the directory of the selected problem. When using cmake (see below), separate executables are built for all of the problem directories listed in Exec/CMakeLists.txt. The problem directories within Exec are sorted into 1) science-relevant setups, such as ABL for modeling the atmospheric boundary layer or DensityCurrent for running the standard density current test case, etc, 2) regression tests in Exec/RegTests that are used for testing specific known aspects of the code functionality, such as boundary conditions or Rayleigh damping, and 3) tests for features under development in Exec/DevTests, such as moving terrain. There is a README in each problem directory that describes the purpose/role of that problem.

GNU Make

The GNU Make system is best for use on large computing facility machines and production runs. With the GNU Make implementation, the build system will inspect the machine and use known compiler optimizations explicit to that machine if possible. These explicit settings are kept up-to-date by the AMReX project.

Using the GNU Make build system involves first setting environment variables for the directories of the dependencies of ERF (AMReX, RTE-RRTMGP, and YAKL); note, RTE-RRTMGP, and YAKL are only required if running with radiation. All dependencies are provided as git submodules in ERF and can be populated by using git submodule init; git submodule update in the ERF repo, or before cloning by using git clone --recursive <erf_repo>. Although submodules of these projects are provided, they can be placed externally as long as the <REPO_HOME> environment variables for each dependency is set correctly. An example of setting the <REPO_HOME> environment variables in the user’s .bashrc is shown below:

export ERF_HOME=${HOME}/ERF
export AMREX_HOME=${ERF_HOME}/Submodules/AMReX

The GNU Make system is set up to use the path to AMReX submodule by default, so it is not necessary to set these paths explicitly, unless it is desired to do so. It is also possible to use an external version of AMReX, downloaded by running

git clone https://github.com/amrex-codes/amrex.git

in which case the AMREX_HOME environment variable must point to the location where AMReX has been downloaded, which will take precedence over the default path to the submodule. If using bash shell,

export AMREX_HOME=/path/to/external/amrex

or if using tcsh,

setenv AMREX_HOME /path/to/external/amrex

  1. cd to the desired build directory, e.g. ERF/Exec/RegTests/IsentropicVortex/

  2. Edit the GNUmakefile; options include

    Option name

    Description

    Possible values

    Default value

    COMP

    Compiler (gnu or intel)

    gnu / intel

    None

    USE_MPI

    Whether to enable MPI

    TRUE / FALSE

    FALSE

    USE_OMP

    Whether to enable OpenMP

    TRUE / FALSE

    FALSE

    USE_CUDA

    Whether to enable CUDA

    TRUE / FALSE

    FALSE

    USE_HIP

    Whether to enable HIP

    TRUE / FALSE

    FALSE

    USE_SYCL

    Whether to enable SYCL

    TRUE / FALSE

    FALSE

    USE_NETCDF

    Whether to enable NETCDF

    TRUE / FALSE

    FALSE

    USE_HDF5

    Whether to enable HDF5

    TRUE / FALSE

    FALSE

    USE_PARTICLES

    Whether to enable particles

    TRUE / FALSE

    FALSE

    USE_WARM_NO_PRECIP

    Whether to use warm moisture

    TRUE / FALSE

    FALSE

    USE_MULTIBLOCK

    Whether to enable multiblock

    TRUE / FALSE

    FALSE

    DEBUG

    Whether to use DEBUG mode

    TRUE / FALSE

    FALSE

    PROFILE

    Include profiling info

    TRUE / FALSE

    FALSE

    TINY_PROFILE

    Include tiny profiling info

    TRUE / FALSE

    FALSE

    COMM_PROFILE

    Include comm profiling info

    TRUE / FALSE

    FALSE

    TRACE_PROFILE

    Include trace profiling info

    TRUE / FALSE

    FALSE

    Note

    Do not set both USE_OMP and USE_CUDA to true.

    Information on using other compilers can be found in the AMReX documentation at https://amrex-codes.github.io/amrex/docs_html/BuildingAMReX.html .

  3. Make the executable by typing

    make

    The name of the resulting executable (generated by the GNUmake system) encodes several of the build characteristics, including dimensionality of the problem, compiler name, and whether MPI and/or OpenMP were linked with the executable. Thus, several different build configurations may coexist simultaneously in a problem folder. For example, the default build in ERF/Exec/RegTests/IsentropicVortex will look like ERF3d.gnu.MPI.ex, indicating that this is a 3-d version of the code, made with COMP=gnu, and USE_MPI=TRUE.

Job info

The build information can be accessed by typing

./ERF*ex --describe

in the directory where the executable has been built.

CMake

CMake is often preferred by developers of ERF; CMake allows for building as well as easy testing and verification of ERF through the use of CTest which is included in CMake.

Compiling with CMake involves an additional configure step before using the make command and it is expected that the user has cloned the ERF repo with the --recursive option or performed git submodule init; git submodule update in the ERF repo to populate its submodules.

ERF provides example scripts for CMake configuration in the /path/to/ERF/Build directory. Once the CMake configure step is done, the make command will build the executable.

An example CMake configure command to build ERF with MPI is listed below:

cmake -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DCMAKE_CXX_COMPILER:STRING=mpicxx \
      -DCMAKE_C_COMPILER:STRING=mpicc \
      -DCMAKE_Fortran_COMPILER:STRING=mpifort \
      .. && make

Typically, a user will create a build directory in the project directory and execute the configuration from said directory (cmake <options> ..) before building. Note that CMake is able to generate makefiles for the Ninja build system as well which will allow for faster building of the executable(s).

Analogous to GNU Make, the list of cmake directives is as follows:

Option name

Description

Possible values

Default value

CMAKE_BUILD_TYPE

Whether to use DEBUG

Release / Debug

Release

ERF_ENABLE_MPI

Whether to enable MPI

TRUE / FALSE

FALSE

ERF_ENABLE_OPENMP

Whether to enable OpenMP

TRUE / FALSE

FALSE

ERF_ENABLE_CUDA

Whether to enable CUDA

TRUE / FALSE

FALSE

ERF_ENABLE_HIP

Whether to enable HIP

TRUE / FALSE

FALSE

ERF_ENABLE_SYCL

Whether to enable SYCL

TRUE / FALSE

FALSE

ERF_ENABLE_NETCDF

Whether to enable NETCDF

TRUE / FALSE

FALSE

ERF_ENABLE_HDF5

Whether to enable HDF5

TRUE / FALSE

FALSE

ERF_ENABLE_PARTICLES

Whether to enable particles

TRUE / FALSE

FALSE

ERF_ENABLE_WARM_NO_PRECIP

Whether to use warm moisture

TRUE / FALSE

FALSE

ERF_ENABLE_MULTIBLOCK

Whether to enable multiblock

TRUE / FALSE

FALSE

ERF_ENABLE_RADIATION

Whether to enable radiation

TRUE / FALSE

FALSE

ERF_ENABLE_TESTS

Whether to enable tests

TRUE / FALSE

FALSE

ERF_ENABLE_FCOMPARE

Whether to enable fcompare

TRUE / FALSE

FALSE

Mac with CMake

Tested with macOS 12.7 (Monterey) using cmake (3.27.8), open-mpi (5.0.0), and pkg-config (0.29.2) installed with the homebrew package manager. HDF5 and NetCDF will be compiled from source. The instructions below should be version agnostic.

HDF5 (tested with v1.14.3)

  1. Download latest source package from hdfgroup.org

  2. Extract source code tar xzf hdf5-<version>.tar.gz

  3. Create build directory cd hdf5-<version> && mkdir build && cd build

  4. Configure for your system ../configure --prefix=/usr/local --enable-parallel

  5. Build make -j8 and sudo make install

NetCDF (tested with v4.9.2)

  1. Download latest source package from ucar.edu

  2. (Optional) install Zstd compression library brew install zstd

  3. Create build directory cd netcdf-c-4.9.2 && mkdir build && cd build

  4. Configure for your system ../configure --enable-parallel CC=mpicc CXX=mpicxx LDFLAGS="-L/opt/homebrew/Cellar/zstd/1.5.5/lib" CPPFLAGS="-I/opt/homebrew/Cellar/zstd/1.5.5/include" (omit the LDFLAGS and CPPFLAGS if you do not have Zstd installed) – note that you may encounter cmake errors if you do not have pkg-config installed

  5. Build make -j8 and sudo make install

ERF (tested with commit 40e64ed35ebc080ad61d08aea828330dfbdbc162)

  1. Get latest source code git clone --recursive git@github.com:erf-model/ERF.git

  2. Create build directory cd ERF && mkdir MyBuild && cd MyBuild

  3. Configure with cmake and build

cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
   -DCMAKE_CXX_COMPILER:STRING=mpicxx \
   -DCMAKE_C_COMPILER:STRING=mpicc \
   -DCMAKE_Fortran_COMPILER:STRING=mpifort \
   -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo \
   -DERF_DIM:STRING=3 \
   -DERF_ENABLE_MPI:BOOL=ON \
   -DERF_ENABLE_TESTS:BOOL=ON \
   -DERF_ENABLE_FCOMPARE:BOOL=ON \
   -DERF_ENABLE_DOCUMENTATION:BOOL=OFF \
   -DERF_ENABLE_NETCDF:BOOL=ON \
   -DERF_ENABLE_HDF5:BOOL=ON \
   -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
   .. && make -j8

Perlmutter (NERSC)

Recall the GNU Make system is best for use on large computing facility machines and production runs. With the GNU Make implementation, the build system will inspect the machine and use known compiler optimizations explicit to that machine if possible. These explicit settings are kept up-to-date by the AMReX project.

For Perlmutter at NERSC, look at the general instructions for building ERF using GNU Make, and then you can initialize your environment by loading these modules:

module load PrgEnv-gnu
module load cudatoolkit

Then build ERF as, for example (specify your own path to the AMReX submodule in ERF/Submodules/AMReX):

make -j 4 COMP=gnu USE_MPI=TRUE USE_OMP=FALSE USE_CUDA=TRUE AMREX_HOME=/global/u2/d/dwillcox/dev-erf/ERF/Submodules/AMReX

Finally, you can prepare your SLURM job script, using the following as a guide:

#!/bin/bash

## specify your allocation (with the _g) and that you want GPU nodes
#SBATCH -A m4106_g
#SBATCH -C gpu

## the job will be named "ERF" in the queue and will save stdout to erf_[job ID].out
#SBATCH -J ERF
#SBATCH -o erf_%j.out

## set the max walltime
#SBATCH -t 10

## specify the number of nodes you want
#SBATCH -N 2

## we use the same number of MPI ranks per node as GPUs per node
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-node=4
#SBATCH --gpu-bind=none

# pin to closest NIC to GPU
export MPICH_OFI_NIC_POLICY=GPU

# use GPU-aware MPI
#GPU_AWARE_MPI=""
GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1"

# the -n argument is (--ntasks-per-node) * (-N) = (number of MPI ranks per node) * (number of nodes)
# set ordering of CUDA visible devices inverse to local task IDs for optimal GPU-aware MPI
srun -n 8 --cpus-per-task=32 --cpu-bind=cores bash -c "
  export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
  ./ERF3d.gnu.MPI.CUDA.ex inputs_wrf_baseline max_step=100 ${GPU_AWARE_MPI}" \
> test.out

To submit your job script, do sbatch [your job script] and you can check its status by doing squeue -u [your username].

Running

The input file specified on the command line is a free-format text file, one entry per row, that specifies input data processed by the AMReX ParmParse module.

This file needs to be specified along with the executable as an argv option, for example:

mpirun -np 64 ./ERF3d.xxx.yyy.ex inputs

Also, any entry that can be specified in the inputs file can also be specified on the command line; values specified on the command line override values in the inputs file, e.g.:

mpirun -np 64 ./ERF3d.gnu.DEBUG.MPI.ex inputs amr.restart=chk0030 erf.use_gravity=true

See Inputs for details on run-time options that can be specified. If running on a Mac and getting errors like SIGILL Invalid, privileged, or ill-formed instruction, see the note on that page about runtime error-checking options.

# To be added later #.. include:: tutorials.rst

Testing and Verification

Testing and verfication of ERF can be performed using CTest, which is included in the CMake build system. If one builds ERF with CMake, the testing suite, and the verification suite, can be enabled during the CMake configure step.

An example cmake configure command performed in the Build directory in ERF is shown below with options relevant to the testing suite:

cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DCMAKE_CXX_COMPILER:STRING=mpicxx \
      -DCMAKE_C_COMPILER:STRING=mpicc \
      -DCMAKE_Fortran_COMPILER:STRING=mpifort \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_USE_CPP:BOOL=ON \
      ..

While performing a cmake -LAH .. command will give descriptions of every option for the CMake project. Descriptions of particular options regarding the testing suite are listed below:

ERF_ENABLE_FCOMPARE – builds the fcompare utility from AMReX as well as the executable(s), to allow for testing differences between plot files

ERF_ENABLE_TESTS – enables the base level regression test suite that will check whether each test will run its executable to completion successfully

Building the Tests

Once the user has performed the CMake configure step, the make command will build every executable required for each test. In this step, it is highly beneficial for the user to use the -j option for make to build source files in parallel.

Running the Tests

Once the test executables are built, CTest also creates working directories for each test within the Build directory where plot files will be output, etc. This directory is analogous to the source location of the tests in Tests/test_files.

To run the test suite, run ctest in the Build directory. CTest will run the tests and report their exit status. Useful options for CTest are -VV which runs in a verbose mode where the output of each test can be seen. -R where a regex string can be used to run specific sets of tests. -j where CTest will bin pack and run tests in parallel based on how many processes each test is specified to use and fit them into the amount of cores available on the machine. -L where the subset of tests containing a particular label will be run. Output for the last set of tests run is available in the Build directory in Tests/Temporary/LastTest.log.

Adding Tests

Developers are encouraged to add tests to ERF and in this section we describe how the tests are organized in the CTest framework. The locations (relative to the ERF code base) of the tests are in Tests. To add a test, first create a problem directory with a name in Exec/RegTests/<prob_name> (for problems to be used as regression tests) or Exec/DevTests/<prob_name> (for problems testing features under development), depending on which type of test is being added. Prepare a suitable input file. As an example, the TaylorGreenVortex problem with input file Exec/RegTests/TaylorGreenVortex/inputs_ex solves a simple advection-diffusion problem. The corresponding regression tests are driven by the input files Tests/test_files/TaylorGreenAdvecting/TaylorGreenAdvecting.i and Tests/test_files/TaylorGreenAdvectingDiffusing/TaylorGreenAdvectingDiffusing.i.

Any file in the test directory will be copied during CMake configure to the test’s working directory. The input files meant for regression test run only until a few time steps. The reference solution that the regression test will refer to should be placed in Tests/ERFGoldFiles/<test_name>. Next, edit the Exec/CMakeLists.txt and Tests/CTestList.cmake files, add the problem and the corresponding tests to the list. Note that there are different categories of tests and if your test falls outside of these categories, a new function to add the test will need to be created. After these steps, your test will be automatically added to the test suite database when doing the CMake configure with the testing suite enabled.