Machine Profiles, Cray Detection, Build Scripts, and Workstation Builds¶

Building ERF on High-Performance Computing (HPC) systems requires managing diverse architectures, proprietary compiler toolchains, and specialized environment modules. ERF automates much of this with machine profile files and the Cray Detection System.

This page focuses on shared build concepts. For basic build instructions, see Build Systems and Options. For initial setup, see Quickstart: Clone-Build-Run. For machine-specific build and run steps, see HPC System Guides.

Note

HPC system environments change regularly. Module names, versions, and availability vary as systems are updated. Always verify current module names with module avail before building. The profile scripts in Build/machines/ are maintained for major systems but may require adjustment after system upgrades.

Note

CMake workflow: The examples below use out-of-source builds (mkdir build && cd build), which keeps build artifacts separate from source code. This is recommended for HPC systems where you may need multiple build configurations or want to preserve a clean source tree. For other workflow options, see Build Systems and Options.

Machine Profile Files¶

Machine profile files prepare the shell environment for compilation by loading required software modules and setting environment variables. They are located in Build/machines/ and provide a standardized, reproducible way to configure the build environment. See https://warpx.readthedocs.io/en/latest/install/hpc.html for detailed examples that include their module load and use environment hints for setting compilers.

Purpose

Profile files execute module load commands and export environment variables. On Cray systems, the core is the PrgEnv-* module (e.g., PrgEnv-gnu or PrgEnv-cray), which dictates the compiler suite that Cray wrappers (cc, CC) will use. This is supplemented by modules for hardware acceleration (e.g., craype-accel-nvidia80), parallel libraries (e.g., cray-netcdf-hdf5parallel), and development tools (e.g., cmake).

Usage

Source the appropriate profile from your shell:

source Build/machines/perlmutter_erf.profile

This modifies the current shell session. The shell is then ready for subsequent build commands.

Available Profiles

ERF provides pre-configured profiles for major DOE HPC systems:

perlmutter_erf.profile - NERSC Perlmutter (NVIDIA A100)
frontier_erf.profile - OLCF Frontier (AMD MI250X)
aurora_erf.profile - ALCF Aurora (Intel GPUs); uses system NetCDF modules (NETCDF_C_ROOT, NETCDF_FORTRAN_ROOT, HDF5_ROOT set by modules) — see also Aurora (ALCF): Build and Run with SYCL
polaris_erf.profile - ALCF Polaris (NVIDIA A100)
kestrel_erf.profile - NREL Kestrel (NVIDIA H100)

Customizing for New Systems

To build on an unsupported HPC system, copy an existing profile for a similar architecture and modify the module load and export commands to match the target system’s software environment.

The Cray Detection System (CMake)¶

The Cray Detection System is a two-phase automation mechanism within ERF’s CMake build system. It identifies Cray Programming Environments on HPC platforms and applies necessary configurations and workarounds, shielding users from platform-specific complexity.

What It Automates

Sets Cray compiler wrappers (cc, CC, ftn) as default compilers
Inspects $CRAY_ACCEL_TARGET to determine GPU architecture (e.g., NVIDIA A100, AMD MI250X)
Automatically sets correct GPU architecture flags for AMReX (AMReX_CUDA_ARCH, AMReX_AMD_ARCH) and Kokkos (Kokkos_ARCH_*)
Configures support for GPU-aware MPI
Finds parallel versions of NetCDF and HDF5

Controlling Auto-Detection

Cray auto-detection is enabled by default when a Cray environment is identified. To disable for manual configuration:

cmake -DERF_ENABLE_CRAY_AUTO_FIXES=OFF ..

Automated Workarounds

The Cray detection system automatically resolves common configuration challenges:

Problem	Detection Clue	Automated Solution
GPU Architecture flags	`$CRAY_ACCEL_TARGET` set (e.g., `nvidia80`)	Maps to correct flags (e.g., `AMReX_CUDA_ARCH=8.0`, `Kokkos_ARCH_AMPERE80=ON`)
CUDA+EKAT MPI paths	`ERF_ENABLE_CUDA=ON` + EKAT physics enabled	Executes `CC --cray-print-opts=cflags` and prepends to `CMAKE_CUDA_FLAGS`
GPU-Aware MPI linking	`$MPICH_GPU_SUPPORT_ENABLED=1`	Links appropriate GTL library (`-lmpi_gtl_cuda` or `-lmpi_gtl_hsa`)
Parallel NetCDF/HDF5	`ERF_ENABLE_NETCDF=ON`	Retrieves Cray search path via `CC --cray-print-opts=PKG_CONFIG_PATH`

Build Scripts Reference¶

ERF provides tested build scripts for common configurations. The following table shows which scripts have been verified on each system.

The test procedure is documented in notes_test.sh.

3 Verified Build Scripts by System¶
Build Script	Perlmutter	Frontier	Aurora	Polaris	Kestrel	RegtestCPU	RegtestGPU
`cmake.sh`	22e12035 (ABL)	22e12035 (ABL)	22e12035	f8665c28 (ABL)	Untested	f8665c28 (ABL)	f8665c28 (ABL)
`cmake_with_kokkos_many.sh`	Untested	Untested	Untested	Untested	Untested	22e12035 (ABL)	Untested
`cmake_with_kokkos_many_cuda.sh`	f8665c28 (ABL)	—	—	Untested	Untested	—	f8665c28 (ABL)
`cmake_with_kokkos_many_noradiation_hip.sh`	—	22e12035 (compiled)	—	—	—	—	—
`cmake_with_kokkos_many_hip.sh`	—	22e12035	—	—	—	—	—
`cmake_with_kokkos_many_sycl.sh`	—	—	22e12035	—	—	—	—
`build_erf_with_shoc.sh`	22e12035	22e12035	22e12035	Untested	Untested	Tested	Tested
`build_erf_with_shoc_cuda.sh`	22e12035	—	—	Untested	Untested	—	Untested
`build_erf_with_shoc_hip.sh`	—	22e12035	—	Untested	Untested	—	Untested
`build_erf_with_shoc_sycl.sh`	—	—	22e12035	Untested	Untested	—	Untested
`Perlmutter/build_erf_with_shoc_cuda_Perlmutter.sh`	Untested	Untested	Untested	Untested	Untested	—	Untested

Note

Reading the table:

Commit hashes link to GitHub and indicate last successful test
(ABL) indicates tested with Atmospheric Boundary Layer case and validated with fcompare
(compiled) indicates build succeeded but runtime results need investigation
Tested means successful build/run without specific commit hash
Untested means not verified but may work with modifications
— indicates configuration not applicable to that system

Script Descriptions

cmake.sh - Basic CPU-only build
cmake_with_kokkos_many.sh - Kokkos-enabled CPU build
cmake_with_kokkos_many_cuda.sh - NVIDIA GPUs with CUDA
cmake_with_kokkos_many_noradiation_hip.sh - AMD GPUs (HIP) without radiation
cmake_with_kokkos_many_hip.sh - AMD GPUs (HIP) full build
cmake_with_kokkos_many_sycl.sh - Intel GPUs with SYCL
build_erf_with_shoc.sh - Automated SHOC workflow (CPU)
build_erf_with_shoc_{cuda,hip,sycl}.sh - Automated SHOC workflow with GPU backend
Perlmutter/build_erf_with_shoc_cuda_Perlmutter.sh - SHOC with CUDA on Perlmutter

Note

The GPU SHOC scripts can be auto-generated. Set BACKEND=CUDA (or HIP/SYCL), then:

sed "/ERF_ENABLE_MPI/a\      -DERF_ENABLE_${BACKEND}:BOOL=ON \\\\" Build/cmake_with_shoc.sh > Build/cmake_with_shoc_${BACKEND,,}.sh
sed "s/cmake_with_shoc.sh/cmake_with_shoc_${BACKEND,,}.sh/" Build/build_erf_with_shoc.sh > Build/build_erf_with_shoc_${BACKEND,,}.sh
chmod +x Build/cmake_with_shoc_${BACKEND,,}.sh Build/build_erf_with_shoc_${BACKEND,,}.sh

Build Script Examples

Basic CPU Build:

#!/bin/bash
# Defaults (customize here or set ERF_BUILD_DIR/ERF_SOURCE_DIR/ERF_INSTALL_DIR in environment)
# If ERF_HOME is set, use it as base for absolute paths
if [ -n "$ERF_HOME" ]; then
  : ${ERF_BUILD_DIR:="$ERF_HOME/build"}
  : ${ERF_SOURCE_DIR:="$ERF_HOME"}
  : ${ERF_INSTALL_DIR:="$ERF_HOME/install"}
else
  : ${ERF_BUILD_DIR:="."}
  : ${ERF_SOURCE_DIR:=".."}
  : ${ERF_INSTALL_DIR:="install"}
fi

echo "Source: $ERF_SOURCE_DIR | Build: $ERF_BUILD_DIR | Install: $ERF_INSTALL_DIR | PWD: $(pwd)"
echo "Customize: export ERF_BUILD_DIR=... ERF_SOURCE_DIR=... ERF_INSTALL_DIR=... or ERF_HOME=..."

cmake -DCMAKE_INSTALL_PREFIX:PATH=$ERF_INSTALL_DIR \
      -DCMAKE_CXX_COMPILER:STRING=mpicxx \
      -DCMAKE_C_COMPILER:STRING=mpicc \
      -DCMAKE_Fortran_COMPILER:STRING=mpifort \
      -DMPIEXEC_PREFLAGS:STRING=--oversubscribe \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_DIM:STRING=3 \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_DOCUMENTATION:BOOL=OFF \
      -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
      --log-context --log-level STATUS \
      -B $ERF_BUILD_DIR -S $ERF_SOURCE_DIR && \
cmake --build $ERF_BUILD_DIR -j10 && \
cmake --install $ERF_BUILD_DIR --prefix=$ERF_INSTALL_DIR

CUDA + Kokkos:

#!/bin/bash
# Defaults (customize here or set ERF_BUILD_DIR/ERF_SOURCE_DIR/ERF_INSTALL_DIR in environment)
# If ERF_HOME is set, use it as base for absolute paths
if [ -n "$ERF_HOME" ]; then
  : ${ERF_BUILD_DIR:="$ERF_HOME/build"}
  : ${ERF_SOURCE_DIR:="$ERF_HOME"}
  : ${ERF_INSTALL_DIR:="$ERF_HOME/install"}
else
  : ${ERF_BUILD_DIR:="."}
  : ${ERF_SOURCE_DIR:=".."}
  : ${ERF_INSTALL_DIR:="install"}
fi

echo "Source: $ERF_SOURCE_DIR | Build: $ERF_BUILD_DIR | Install: $ERF_INSTALL_DIR | PWD: $(pwd)"
echo "Customize: export ERF_BUILD_DIR=... ERF_SOURCE_DIR=... ERF_INSTALL_DIR=... or ERF_HOME=..."

cmake -DCMAKE_INSTALL_PREFIX:PATH=$ERF_INSTALL_DIR \
      -DMPIEXEC_PREFLAGS:STRING=--oversubscribe \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_DIM:STRING=3 \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DERF_ENABLE_CUDA:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_ENABLE_ALL_WARNINGS:BOOL=ON \
      -DERF_ENABLE_RRTMGP:BOOL=ON \
      -DERF_ENABLE_NETCDF:BOOL=ON \
      -DERF_ENABLE_HDF5:BOOL=ON \
      -DERF_ENABLE_FFT:BOOL=ON \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_DOCUMENTATION:BOOL=OFF \
      -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
      --log-context --log-level STATUS \
      -B $ERF_BUILD_DIR -S $ERF_SOURCE_DIR && \
cmake --build $ERF_BUILD_DIR -j10 && \
cmake --install $ERF_BUILD_DIR --prefix=$ERF_INSTALL_DIR

HIP + Kokkos:

#!/bin/bash
# Defaults (customize here or set ERF_BUILD_DIR/ERF_SOURCE_DIR/ERF_INSTALL_DIR in environment)
# If ERF_HOME is set, use it as base for absolute paths
if [ -n "$ERF_HOME" ]; then
  : ${ERF_BUILD_DIR:="$ERF_HOME/build"}
  : ${ERF_SOURCE_DIR:="$ERF_HOME"}
  : ${ERF_INSTALL_DIR:="$ERF_HOME/install"}
else
  : ${ERF_BUILD_DIR:="."}
  : ${ERF_SOURCE_DIR:=".."}
  : ${ERF_INSTALL_DIR:="install"}
fi

echo "Source: $ERF_SOURCE_DIR | Build: $ERF_BUILD_DIR | Install: $ERF_INSTALL_DIR | PWD: $(pwd)"
echo "Customize: export ERF_BUILD_DIR=... ERF_SOURCE_DIR=... ERF_INSTALL_DIR=... or ERF_HOME=..."

cmake -DCMAKE_INSTALL_PREFIX:PATH=$ERF_INSTALL_DIR \
      -DMPIEXEC_PREFLAGS:STRING=--oversubscribe \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_DIM:STRING=3 \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DERF_ENABLE_HIP:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_ENABLE_ALL_WARNINGS:BOOL=ON \
      -DERF_ENABLE_RRTMGP:BOOL=ON \
      -DERF_ENABLE_NETCDF:BOOL=ON \
      -DERF_ENABLE_HDF5:BOOL=ON \
      -DERF_ENABLE_FFT:BOOL=ON \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_DOCUMENTATION:BOOL=OFF \
      -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
      --log-context --log-level STATUS \
      -B $ERF_BUILD_DIR -S $ERF_SOURCE_DIR && \
cmake --build $ERF_BUILD_DIR -j10 && \
cmake --install $ERF_BUILD_DIR --prefix=$ERF_INSTALL_DIR

SYCL + Kokkos:

#!/bin/bash
# Defaults (customize here or set ERF_BUILD_DIR/ERF_SOURCE_DIR/ERF_INSTALL_DIR in environment)
# If ERF_HOME is set, use it as base for absolute paths
if [ -n "$ERF_HOME" ]; then
  : ${ERF_BUILD_DIR:="$ERF_HOME/build"}
  : ${ERF_SOURCE_DIR:="$ERF_HOME"}
  : ${ERF_INSTALL_DIR:="$ERF_HOME/install"}
else
  : ${ERF_BUILD_DIR:="."}
  : ${ERF_SOURCE_DIR:=".."}
  : ${ERF_INSTALL_DIR:="install"}
fi

echo "Source: $ERF_SOURCE_DIR | Build: $ERF_BUILD_DIR | Install: $ERF_INSTALL_DIR | PWD: $(pwd)"
echo "Customize: export ERF_BUILD_DIR=... ERF_SOURCE_DIR=... ERF_INSTALL_DIR=... or ERF_HOME=..."

cmake -DCMAKE_INSTALL_PREFIX:PATH=$ERF_INSTALL_DIR \
      -DMPIEXEC_PREFLAGS:STRING=--oversubscribe \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_DIM:STRING=3 \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DERF_ENABLE_SYCL:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_ENABLE_ALL_WARNINGS:BOOL=ON \
      -DERF_ENABLE_RRTMGP:BOOL=ON \
      -DERF_ENABLE_NETCDF:BOOL=ON \
      -DERF_ENABLE_HDF5:BOOL=ON \
      -DERF_ENABLE_FFT:BOOL=ON \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_DOCUMENTATION:BOOL=OFF \
      -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
      --log-context --log-level STATUS \
      -B $ERF_BUILD_DIR -S $ERF_SOURCE_DIR && \
cmake --build $ERF_BUILD_DIR -j10 && \
cmake --install $ERF_BUILD_DIR --prefix=$ERF_INSTALL_DIR

SHOC (CPU):

#!/bin/bash
# Defaults (customize here or set ERF_BUILD_DIR/ERF_SOURCE_DIR/ERF_INSTALL_DIR in environment)
# If ERF_HOME is set, use it as base for absolute paths
if [ -n "$ERF_HOME" ]; then
  : ${ERF_BUILD_DIR:="$ERF_HOME/build"}
  : ${ERF_SOURCE_DIR:="$ERF_HOME"}
  : ${ERF_INSTALL_DIR:="$ERF_HOME/install"}
else
  : ${ERF_BUILD_DIR:="."}
  : ${ERF_SOURCE_DIR:=".."}
  : ${ERF_INSTALL_DIR:="install"}
fi

echo "Source: $ERF_SOURCE_DIR | Build: $ERF_BUILD_DIR | Install: $ERF_INSTALL_DIR | PWD: $(pwd)"
echo "Customize: export ERF_BUILD_DIR=... ERF_SOURCE_DIR=... ERF_INSTALL_DIR=... or ERF_HOME=..."

cmake -DCMAKE_INSTALL_PREFIX:PATH=$ERF_INSTALL_DIR \
      -DMPIEXEC_PREFLAGS:STRING=--oversubscribe \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_DIM:STRING=3 \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_ENABLE_SHOC:BOOL=ON \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_DOCUMENTATION:BOOL=OFF \
      -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
      --log-context --log-level STATUS \
      -B $ERF_BUILD_DIR -S $ERF_SOURCE_DIR && \
cmake --build $ERF_BUILD_DIR -j10 && \
cmake --install $ERF_BUILD_DIR --prefix=$ERF_INSTALL_DIR

Requires E3SM submodules: source Build/GNU_Ekat/eamxx_clone.sh

Automated SHOC Workflow:

#!/bin/bash
set -e
set -o pipefail

# Function to verify if a directory is the ERF repo root
verify_erf_dir() {
    local dir=$1

    # Check for basic structure
    if [ ! -f "$dir/CMakeLists.txt" ] || [ ! -d "$dir/Source" ]; then
        return 1
    fi

    # Check for "Energy Research and Forecasting" in key files
    local found=0

    if [ -f "$dir/README.rst" ]; then
        if grep -q "Energy Research and Forecasting" "$dir/README.rst" 2>/dev/null; then
            found=1
        fi
    fi

    if [ $found -eq 0 ] && [ -f "$dir/LICENSE.md" ]; then
        if grep -q "Energy Research and Forecasting" "$dir/LICENSE.md" 2>/dev/null; then
            found=1
        fi
    fi

    if [ $found -eq 0 ] && [ -f "$dir/CITATION.cff" ]; then
        if grep -q "Energy Research and Forecasting" "$dir/CITATION.cff" 2>/dev/null; then
            found=1
        fi
    fi

    return $((1 - found))
}

# Function to find ERF repo root with multiple fallbacks
find_erf_dir() {
    # Method 1: Use git to find repo root
    if command -v git &> /dev/null; then
        if git rev-parse --is-inside-work-tree &> /dev/null 2>&1; then
            local git_root="$(git rev-parse --show-toplevel)"
            if verify_erf_dir "$git_root"; then
                ERF_DIR="$git_root"
                echo "Detected ERF_DIR from git: $ERF_DIR"
                return 0
            fi
        fi
    fi

    # Method 2: Try going up from script location
    local script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
    # Script is in Build/Perlmutter/, so go up 2 levels
    local candidate="$(cd "$script_dir/../.." && pwd)"
    if verify_erf_dir "$candidate"; then
        ERF_DIR="$candidate"
        echo "Detected ERF_DIR from script location: $ERF_DIR"
        return 0
    fi

    # Method 3: Check current directory
    if verify_erf_dir "$PWD"; then
        ERF_DIR="$PWD"
        echo "Detected ERF_DIR from current directory: $ERF_DIR"
        return 0
    fi

    echo "Error: Could not auto-detect ERF_DIR"
    echo "Verification requires:"
    echo "  - CMakeLists.txt and Source/ directory"
    echo "  - 'Energy Research and Forecasting' in README.rst, LICENSE.md, or CITATION.cff"
    return 1
}

###################################################################################

# 1. Resolve ERF_DIR
# Detect ERF_DIR
if ! find_erf_dir; then
    exit 1
fi

export ERF_DIR

E3SM_DIR="$ERF_DIR/external/E3SM"
if [ ! -d "$E3SM_DIR" ]; then
    echo "external/E3SM folder not found, running eamxx_clone.sh..."
    source "$ERF_DIR/Build/GNU_Ekat/eamxx_clone.sh"
else
    echo "external/E3SM folder already exists, skipping clone."
fi

# 3. Prepare build directory
echo "Preparing build directory..."
mkdir -p "$ERF_DIR/build"
cp "$ERF_DIR/Build/cmake_with_shoc.sh" "$ERF_DIR/build/"

# 4. Move into build directory
cd "$ERF_DIR/build"

# Run cmake setup
echo "Running cmake_with_shoc.sh..."
source cmake_with_shoc.sh

Usage: source Build/build_erf_with_shoc.sh

For detailed CMake options and dependencies, see Build Systems and Options.

GNU Make vs CMake on HPC

Both build systems are fully supported. The choice depends on desired abstraction level and control.

Build System	Strengths & Best Use Cases
CMake	Strengths: Automates complex dependency graphs (ERF → EKAT → Kokkos). Cray Detection System handles configurations. Generates `find_package`-compatible configuration (`ERFConfig.cmake`). Best For: Most users on supported HPC systems (Perlmutter, Frontier, Polaris, Aurora). Developers building applications that depend on ERF.
GNU Make	Strengths: Uses existing AMReX infrastructure. Utility targets (e.g., `print-xxx`) for debugging. Direct control over every build variable. Best For: Advanced users, performance tuning experts, systems not yet supported by CMake auto-detection.

Workstation Builds¶

Building on a local workstation is simpler than on HPC systems, as it doesn’t require managing environment modules or vendor-specific toolchains. This is essential for development, debugging, and running smaller test cases.

The RegtestCPU and RegtestGPU columns in the build scripts table indicate configurations tested on workstation environments based on the nightly regression test setups.

macOS Builds

Building on macOS is supported with specific considerations:

GNU Make is often recommended for simplicity and direct control
Use Make.local (amrex/Tools/GNUMake/Make.local) to specify consistent compiler suite (e.g., GCC from Homebrew) for both C++ and Fortran
This avoids conflicts between default Clang/Xcode C++ compiler and separately installed Fortran compiler

Example Make.local for macOS:

CXX = g++-13
CC  = gcc-13
FC  = gfortran-13