Build Systems and Options¶

This document explains ERF’s two build systems, GNU Make and CMake, and how each manages dependencies to produce a functional executable.

The two build systems serve distinct purposes:

GNU Make - Application-centric system for compiling a single executable for a specific scientific run. Provides fine-grained control over compilation and debugging via utility targets like print-xxx.
CMake - Framework-centric system designed for cross-platform compatibility and dependency management. Creates versioned libraries, provides automated detection (including Cray systems), and supports testing through CTest. Recommended for HPC environments.

This guide serves as a technical reference for developers and advanced users. For initial setup and quick start instructions, see Quickstart: Clone-Build-Run and Getting Started. For HPC concepts, see Machine Profiles, Cray Detection, Build Scripts, and Workstation Builds; for machine-specific build/run workflows, see HPC System Guides.

Directory Structure and Workflow¶

ERF builds executables in ERF/Exec when using GNU Make. The exception is development-test work under ERF/.Exec_dev, where GNU Make builds in a problem-specific directory under ERF/.Exec_dev.

With CMake, one configures once to build the core libraries and the shared test executable in build (e.g., erf_exec).

Regardless of how the executable is built, it can be run on regression and canonical tests that have inputs files in subdirectories of Exec/RegTests and Exec/CanonicalTests. Ctests are run on files in subdirectories of ERF/Tests/test_files.

Directories within Exec are organized into cases used primarily for regression testing and cases representing more canonical tests.

Exec/
├── RegTests/               # Regression test input decks
│   ├── IsentropicVortex/
│   ├── TaylorGreenVortex/
│   ├── Bubble/
│   └── ...
├── CanonicalTests/         # Canonical atmospheric flow input decks
    ├── ABL/
    ├── Bomex/
    ├── SquallLine/
    ├── SuperCell/
    └── ...

Each problem directory contains a README describing its purpose and functionality.

Building with GNU Make¶

System Overview¶

The GNU Make system provides a direct path to producing a single, non-case-specific executable that works for all cases represented in ERF/Exec/RegTests and ERF/Exec/CanonicalTests.

Depending on the cases of interest, the user must specify in ERF/Exec/GNUmakefile whether the executable should be built with particle capability (USE\_PARTICLES = TRUE), with NetCDF capability (USE\_NETCDF = TRUE), with radiation (USE\_RRGMTP = TRUE), and several other compile-time options.

The build is orchestrated by a GNUmakefile in ERF/Exec/ (and similarly in ERF/.Exec_dev for development tests), which uses build logic from the AMReX framework.

Build Steps¶

1. Clone Repository and Populate Dependencies

See Git Submodules for more details. All dependencies except SHOC and P3 are provided as git submodules:

# Clone with submodules
git clone --recursive https://github.com/erf-model/ERF.git

# Or populate submodules in existing clone
git submodule update --init --recursive

The GNU Make system uses the AMReX submodule path by default. To use an external AMReX installation, set AMREX_HOME:

# Download external AMReX
git clone https://github.com/amrex-codes/amrex.git

# Set environment variable (bash)
export AMREX_HOME=/path/to/external/amrex

# Or for tcsh
setenv AMREX_HOME /path/to/external/amrex

2. Setup for SHOC or P3 (Optional)

If building with SHOC or P3, run the setup scripts:

export ERF_DIR=/path/to/ERF
source /path/to/ERF/Build/GNU_Ekat/eamxx_clone.sh
source /path/to/ERF/Build/GNU_Ekat/ekat_build_commands.sh

Then set USE_SHOC=TRUE or USE_P3=TRUE in your GNUmakefile (step 4).

3. Navigate to GNU Make Build Directory

# Standard workflow
cd ERF/Exec

# Development-test workflow (exception)
# cd ERF/.Exec_dev/<test_name>

4. Edit GNUmakefile

Set build variables in the GNUmakefile:

1 GNU Make Build Variables¶

Variable Name

Description

Default

Possible Values

AMREX_HOME

Specifies path to AMReX source directory

$(ERF_HOME)/Submodules/AMReX

Path string

COMP

Defines compiler suite to use

None (required)

gnu/intel/cray

DIM

Sets dimensionality of problem (ERF is 3D only)

3

3

USE_MPI

Enables MPI for distributed-memory parallel execution

FALSE

TRUE/FALSE

USE_OMP

Enables OpenMP for shared-memory parallelism

FALSE

TRUE/FALSE

USE_CUDA

Enables NVIDIA GPU support via CUDA

FALSE

TRUE/FALSE

USE_HIP

Enables AMD GPU support via HIP

FALSE

TRUE/FALSE

USE_SYCL

Enables Intel GPU support via SYCL

FALSE

TRUE/FALSE

USE_NETCDF

Enables I/O support using NetCDF library

FALSE

TRUE/FALSE

USE_PARTICLES

Enables support for Lagrangian particles

FALSE

TRUE/FALSE

USE_NOAHMP

Enables Noah-MP land surface model (requires USE_NETCDF=TRUE)

FALSE

TRUE/FALSE

USE_RRTMGP

Enables RRTMGP radiation model (sets USE_KOKKOS=TRUE, USE_NETCDF=TRUE)

FALSE

TRUE/FALSE

USE_SHOC

Enables SHOC turbulence model (sets USE_KOKKOS=TRUE)

FALSE

TRUE/FALSE

USE_P3

Enables P3 microphysics model (sets USE_KOKKOS=TRUE)

FALSE

TRUE/FALSE

USE_MULTIBLOCK

Enables multiblock capability

FALSE

TRUE/FALSE

USE_KOKKOS

Enables Kokkos performance portability library

FALSE

TRUE/FALSE

USE_MORR_FORT

Enables Fortran-based Morrison microphysics scheme

FALSE

TRUE/FALSE

USE_FFT

Enables Fast Fourier Transform capabilities

FALSE

TRUE/FALSE

DEBUG

Enables debug mode

FALSE

TRUE/FALSE

PROFILE

Includes profiling info

FALSE

TRUE/FALSE

TINY_PROFILE

Includes tiny profiling info

FALSE

TRUE/FALSE

COMM_PROFILE

Includes comm profiling info

FALSE

TRUE/FALSE

TRACE_PROFILE

Includes trace profiling info

FALSE

TRUE/FALSE

Note

At most one of USE_OMP, USE_CUDA, USE_HIP, USE_SYCL should be TRUE.

For additional compiler options, see the AMReX documentation.

5. Build

make

The executable name encodes build characteristics (dimensionality, compiler, parallelization). For example, in Exec with COMP=gnu and USE_MPI=TRUE, the executable is ERF3d.gnu.MPI.ex. Multiple build configurations can coexist in the same directory.

6. Verify Build (Optional)

View build configuration:

./ERF*ex --describe

Building Documentation¶

Build ERF documentation with GNU Make:

cd ERF/Docs/
source BuildDocs.sh

This builds both Sphinx and Doxygen documentation.

Building with CMake¶

System Overview¶

CMake is a cross-platform build tool using a framework-centric approach. Instead of producing a single executable for one use case, CMake produces versioned, exportable libraries as well as multiple executables across the problem directory setups. This enables generation of find_package-compatible configuration files (ERFConfig.cmake), allowing other CMake projects to consume ERF as a dependency.

CMake provides:

Out-of-source builds keeping artifacts separate from source code
Robust dependency detection through find_package
Automated detection and configuration for Cray HPC systems
Testing and verification through CTest

CMake is recommended for HPC environments where automated dependency detection and robust cross-platform builds are important.

Build Steps¶

1. Clone Repository and Prerequisites

Clone with submodules:

# Clone with submodules
git clone --recursive https://github.com/erf-model/ERF.git

# Or populate submodules in existing clone
git submodule update --init --recursive

2. Setup for SHOC or P3 (Optional)

If building with SHOC or P3:

export ERF_DIR=/path/to/ERF
source /path/to/ERF/Build/GNU_Ekat/eamxx_clone.sh

Then configure with -DERF_ENABLE_SHOC=TRUE and/or -DERF_ENABLE_P3=TRUE (step 4).

3. Choose Build Workflow

ERF supports multiple CMake workflows. The main difference is directory structure and whether you use the install step.

In Build/ Directory

Build artifacts and executables in the Build/ directory tree. Executables appear in Exec/<problem>/ subdirectories, similar to GNU Make workflow.

cd Build
./cmake.sh

Executable locations: Build/Exec/erf_exec

Cleanup for rebuild:

make distclean
./cmake.sh

Out-of-Source Build

Build directory separate from source. Executables in build/Exec/<problem>/ subdirectories. Optionally use install step to copy all executables to a single install/bin/ directory.

mkdir build && cd build
../Build/cmake.sh
make install  # optional - copies to install/bin/ (may be needed for builds that require kokkos)

Executable locations: build/Exec/erf_exec, etc., and optionally install/bin/erf_exec (if installed)

Cleanup for rebuild:

rm -rf build/ install/  # complete cleanup
mkdir build && cd build

Script with Customizable Directories

Uses environment variables for directory control. Defaults: build in current dir (.), source from parent (..), install to install/.

Customize directories (optional):

Set ERF_BUILD_DIR, ERF_SOURCE_DIR, ERF_INSTALL_DIR, or ERF_HOME environment variables.

# From ERF repository root (sets absolute paths)
ERF_HOME=$(pwd) ./Build/cmake_with_kokkos_many.sh

# Or customize individual directories
ERF_BUILD_DIR=build ERF_INSTALL_DIR=install ./Build/cmake_with_kokkos_many.sh

# Or use defaults from any directory
cd ERF
./Build/cmake_with_kokkos_many.sh

Executable locations: $ERF_BUILD_DIR/Exec/erf_exec and $ERF_INSTALL_DIR/bin/erf_exec (defaults: ./Exec/erf_exec and install/bin/erf_exec)

Cleanup for rebuild:

# If using defaults
rm -rf . install/  # from build directory

# If using ERF_HOME
rm -rf $ERF_HOME/build $ERF_HOME/install

# Then rebuild
./Build/cmake_with_kokkos_many.sh

4. Configure with Options

For manual configuration, use -D<VARIABLE>=<VALUE> syntax:

cmake -DCMAKE_BUILD_TYPE=Release \
      -DERF_ENABLE_MPI=ON \
      -DCMAKE_CXX_COMPILER=mpicxx \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_Fortran_COMPILER=mpifort \
      .. && make

CMake can also generate makefiles for the Ninja build system for faster compilation.

CMake Options:

2 ERF CMake Options¶
Variable Name	Description	Default	Possible Values
`CMAKE_BUILD_TYPE`	Sets build configuration	Release	Release/Debug/RelWithDebInfo
`ERF_ENABLE_MPI`	Enables MPI support for parallel execution	OFF	ON/OFF
`ERF_ENABLE_OPENMP`	Enables OpenMP for shared-memory parallelism	OFF	ON/OFF
`ERF_ENABLE_CUDA`	Enables NVIDIA GPU support via CUDA (requires Toolkit >= 11.0)	OFF	ON/OFF
`ERF_ENABLE_HIP`	Enables AMD GPU support via HIP	OFF	ON/OFF
`ERF_ENABLE_SYCL`	Enables Intel GPU support via SYCL	OFF	ON/OFF
`ERF_ENABLE_NETCDF`	Enables NetCDF for I/O operations	OFF	ON/OFF
`ERF_ENABLE_PARTICLES`	Enables support for Lagrangian particles	OFF	ON/OFF
`ERF_PRECISION`	Floating-point precision for mesh data	DOUBLE	SINGLE/DOUBLE
`ERF_PARTICLES_PRECISION`	Floating-point precision for particle data; defaults to `ERF_PRECISION`	DOUBLE	SINGLE/DOUBLE
`ERF_ENABLE_MULTIBLOCK`	Enables multiblock capability	OFF	ON/OFF
`ERF_ENABLE_NOAHMP`	Enables Noah-MP land surface model (requires `ERF_ENABLE_NETCDF=ON`)	OFF	ON/OFF
`ERF_ENABLE_RRTMGP`	Enables RRTMGP radiation model (requires `ERF_ENABLE_NETCDF=ON`, `ERF_ENABLE_MPI=ON`)	OFF	ON/OFF
`ERF_ENABLE_SHOC`	Enables SHOC turbulence model (requires `ERF_ENABLE_MPI=ON`)	OFF	ON/OFF
`ERF_ENABLE_P3`	Enables P3 microphysics model (requires `ERF_ENABLE_MPI=ON`)	OFF	ON/OFF
`ERF_ENABLE_TESTS`	Enables CTest test suite	OFF	ON/OFF
`ERF_ENABLE_FCOMPARE`	Enables fcompare utility for regression testing	OFF	ON/OFF
`ERF_ENABLE_DOCUMENTATION`	Enables build target for generating Sphinx documentation	OFF	ON/OFF
`CMAKE_INSTALL_PREFIX`	Directory where `make install` places compiled artifacts	System-dependent	Path string

Note

At most one of ERF_ENABLE_OPENMP, ERF_ENABLE_CUDA, ERF_ENABLE_HIP, ERF_ENABLE_SYCL should be ON.

Developer Testing

For systematic testing of multiple build configurations:

#!/bin/bash

set -e
set -o pipefail

# Function to verify if a directory is the ERF repo root
verify_erf_dir() {
    local dir=$1

    # Check for basic structure
    if [ ! -f "$dir/CMakeLists.txt" ] || [ ! -d "$dir/Source" ]; then
        return 1
    fi

    # Check for "Energy Research and Forecasting" in key files
    local found=0

    if [ -f "$dir/README.rst" ]; then
        if grep -q "Energy Research and Forecasting" "$dir/README.rst" 2>/dev/null || true; then
            found=1
        fi
    fi

    if [ $found -eq 0 ] && [ -f "$dir/LICENSE.md" ]; then
        if grep -q "Energy Research and Forecasting" "$dir/LICENSE.md" 2>/dev/null || true; then
            found=1
        fi
    fi

    if [ $found -eq 0 ] && [ -f "$dir/CITATION.cff" ]; then
        if grep -q "Energy Research and Forecasting" "$dir/CITATION.cff" 2>/dev/null || true; then
            found=1
        fi
    fi

    return $((1 - found))
}

# Function to find ERF repo root
find_erf_dir() {
    # Method 1: Check if we're already in Build/
    if [ -f "../CMakeLists.txt" ] && [ -d "../Source" ]; then
        local candidate="$(cd .. && pwd)"
        if verify_erf_dir "$candidate"; then
            ERF_DIR="$candidate"
            echo "Detected ERF_DIR from Build location: $ERF_DIR"
            return 0
        fi
    fi

    # Method 2: Use git to find repo root
    if command -v git &> /dev/null; then
        if git rev-parse --is-inside-work-tree &> /dev/null 2>&1; then
            local git_root="$(git rev-parse --show-toplevel)"
            if verify_erf_dir "$git_root"; then
                ERF_DIR="$git_root"
                echo "Detected ERF_DIR from git: $ERF_DIR"
                return 0
            fi
        fi
    fi

    # Method 3: Try going up from script location
    local script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
    # Check if script is in Build/ directory
    if [[ "$script_dir" =~ /Build$ ]]; then
        local candidate="$(dirname "$script_dir")"
        if verify_erf_dir "$candidate"; then
            ERF_DIR="$candidate"
            echo "Detected ERF_DIR from script location: $ERF_DIR"
            return 0
        fi
    fi

    # Method 4: Check current directory
    if verify_erf_dir "$PWD"; then
        ERF_DIR="$PWD"
        echo "Detected ERF_DIR from current directory: $ERF_DIR"
        return 0
    fi

    return 1
}

# Parse arguments
if [ $# -lt 1 ] || [ $# -gt 3 ]; then
    echo "Usage: $0 <set> [script_pattern] [erf_dir]"
    echo ""
    echo "Sets:"
    echo "  default    - Scripts from Build/"
    echo "  perlmutter - Scripts from Build/Perlmutter/"
    echo "  gnu_ekat   - Scripts from Build/GNU_Ekat/"
    echo ""
    echo "If script_pattern is provided, creates build_<pattern>/"
    echo "Otherwise creates build_<set>/"
    echo ""
    echo "If erf_dir is provided, uses that as ERF_DIR"
    echo "Otherwise auto-detects ERF repo root"
    exit 1
fi

SET=$1
PATTERN=${2:-}
ERF_DIR_ARG=${3:-}

# Set ERF_DIR
if [ -n "$ERF_DIR_ARG" ]; then
    ERF_DIR="$ERF_DIR_ARG"
    echo "Using provided ERF_DIR: $ERF_DIR"
    if ! verify_erf_dir "$ERF_DIR"; then
        echo "Error: Provided directory is not a valid ERF repository"
        echo "Must contain 'Energy Research and Forecasting' in README.rst, LICENSE.md, or CITATION.cff"
        exit 1
    fi
else
    if ! find_erf_dir; then
        echo "Error: Could not auto-detect ERF_DIR"
        echo "Please provide it as the third argument or run from ERF Build/ directory"
        echo ""
        echo "Verification checks for:"
        echo "  - CMakeLists.txt and Source/ directory"
        echo "  - 'Energy Research and Forecasting' in README.rst, LICENSE.md, or CITATION.cff"
        exit 1
    fi
fi

echo "ERF_DIR set to: $ERF_DIR"

# Define source directories relative to ERF_DIR
DEFAULT_DIR="$ERF_DIR/Build"
PERLMUTTER_DIR="$ERF_DIR/Build/Perlmutter"
GNU_EKAT_DIR="$ERF_DIR/Build/GNU_Ekat"

case $SET in
    default)
        SRC_DIR="$DEFAULT_DIR"
        ;;
    perlmutter)
        SRC_DIR="$PERLMUTTER_DIR"
        ;;
    gnu_ekat)
        SRC_DIR="$GNU_EKAT_DIR"
        ;;
    *)
        echo "Error: Invalid set '$SET'"
        echo "Choose: default, perlmutter, or gnu_ekat"
        exit 1
        ;;
esac

if [ ! -d "$SRC_DIR" ]; then
    echo "Error: Source directory does not exist: $SRC_DIR"
    exit 1
fi

# Determine build directory name
if [ -n "$PATTERN" ]; then
    BUILD_DIR="$ERF_DIR/build_${PATTERN}"
else
    BUILD_DIR="$ERF_DIR/build_${SET}"
fi

# Create build directory
mkdir -p "$BUILD_DIR"
echo "Created directory: $BUILD_DIR"

# Find and copy ERF cmake build scripts
echo "Scanning for ERF cmake scripts in $SRC_DIR:"
COPIED=0
SKIPPED=0

# Temporarily disable exit on error for the loop
set +e

for script in "$SRC_DIR"/*.sh; do
    # Check if file exists (glob might not match anything)
    if [ ! -f "$script" ]; then
        continue
    fi

    basename_script=$(basename "$script")

    # Skip backup files
    if [[ "$basename_script" =~ ~$ ]]; then
        SKIPPED=$((SKIPPED + 1))
        continue
    fi

    # Check if it's an ERF cmake script (contains DERF or cmake)
    has_derf=0
    has_cmake=0

    grep -q "DERF" "$script" 2>/dev/null && has_derf=1
    grep -q "cmake" "$script" 2>/dev/null && has_cmake=1

    if [ $has_derf -eq 1 ] || [ $has_cmake -eq 1 ]; then
        cp "$script" "$BUILD_DIR/"
        chmod +x "$BUILD_DIR/$basename_script"
        echo "   DONE: $basename_script"
        COPIED=$((COPIED + 1))
    else
        echo "  ERROR: $basename_script (no DERF or cmake found)"
        SKIPPED=$((SKIPPED + 1))
    fi
done

# Re-enable exit on error
set -e

echo ""
echo "Summary: Copied $COPIED script(s), skipped $SKIPPED"

if [ $COPIED -eq 0 ]; then
    echo "Warning: No ERF cmake scripts found"
    echo "Scripts should contain 'DERF' or 'cmake'"
fi

# Create a run script in the build directory
cat > "$BUILD_DIR/run.sh" << 'EOF'
#!/bin/bash

set -e
set -o pipefail

# Resolve ERF_DIR (go up from build directory)
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
export ERF_DIR="$(dirname "$SCRIPT_DIR")"

echo "ERF_DIR set to: $ERF_DIR"

# Find all .sh scripts (excluding run.sh and backups)
SCRIPTS=()
for script in *.sh; do
    if [ "$script" = "run.sh" ]; then
        continue
    fi
    if [[ "$script" =~ ~$ ]]; then
        continue
    fi
    if [ -f "$script" ]; then
        SCRIPTS+=("$script")
    fi
done

# Sort scripts alphabetically
IFS=$'\n' SCRIPTS=($(sort <<<"${SCRIPTS[*]}"))
unset IFS

if [ ${#SCRIPTS[@]} -eq 0 ]; then
    echo "Error: No build scripts found in this directory"
    exit 1
fi

if [ $# -ne 1 ]; then
    echo "Usage: $0 <number>"
    echo ""
    echo "Available ERF cmake scripts:"
    for i in "${!SCRIPTS[@]}"; do
        script_base="${SCRIPTS[$i]%.sh}"
        printf "%3d: %s\n" $((i+1)) "${SCRIPTS[$i]}"
        printf "     -> subdirectory: %s/script_%s/\n" "$ERF_DIR" "$script_base"
    done
    echo ""
    echo "Each script will run in its own clean subdirectory at ERF root."
    exit 1
fi

NUM=$1
if [ $NUM -lt 1 ] || [ $NUM -gt ${#SCRIPTS[@]} ]; then
    echo "Error: Number must be between 1 and ${#SCRIPTS[@]}"
    exit 1
fi

SCRIPT="${SCRIPTS[$((NUM-1))]}"
SCRIPT_BASE="${SCRIPT%.sh}"
SUBDIR="$ERF_DIR/script_${SCRIPT_BASE}"

# Create a clean subdirectory for this script
if [ -d "$SUBDIR" ]; then
    echo "Warning: $SUBDIR already exists"
    read -p "Delete and recreate? (y/N) " -n 1 -r
    echo
    if [[ $REPLY =~ ^[Yy]$ ]]; then
        rm -rf "$SUBDIR"
    else
        echo "Aborting. Please remove $SUBDIR manually or choose a different script."
        exit 1
    fi
fi

mkdir -p "$SUBDIR"
echo "========================================"
echo "Running: $SCRIPT"
echo "Build directory: $SUBDIR"
echo "Working directory: $SUBDIR"
echo "========================================"
echo ""

# Copy the script into the subdirectory and run it there
cp "$SCRIPT" "$SUBDIR/"
cd "$SUBDIR"
bash "./$SCRIPT"
EOF

chmod +x "$BUILD_DIR/run.sh"

echo ""
echo "Setup complete!"
echo "Build directory: $BUILD_DIR"
echo ""
echo "To use:"
echo "  cd $BUILD_DIR"
echo "  ./run.sh           # List available scripts"
echo "  ./run.sh <number>  # Run a specific script"
echo ""
echo "Copied $COPIED script(s)"

Usage:

cd Build
./setup_cmake_validation.sh default    # or perlmutter, gnu_ekat
cd ../build_default
./run.sh     # List available scripts
./run.sh 1   # Run first script

Creates isolated build directories for each configuration script. Each script runs in its own subdirectory (script_<name>/) at the ERF root.

Building Documentation¶

Build ERF documentation with CMake:

cmake -DERF_ENABLE_DOCUMENTATION=ON ..
make docs

This builds both Sphinx and Doxygen documentation.