Testing and Verification

Testing and verification of ERF can be performed using CTest, which is included in the CMake build system. If one builds ERF with CMake, the testing suite, and the verification suite, can be enabled during the CMake configure step.

An example cmake configure command performed in the build directory in ERF is shown below with options relevant to the testing suite:

cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
      -DCMAKE_BUILD_TYPE:STRING=Release \
      -DERF_ENABLE_MPI:BOOL=ON \
      -DCMAKE_CXX_COMPILER:STRING=mpicxx \
      -DCMAKE_C_COMPILER:STRING=mpicc \
      -DCMAKE_Fortran_COMPILER:STRING=mpifort \
      -DERF_ENABLE_FCOMPARE:BOOL=ON \
      -DERF_ENABLE_TESTS:BOOL=ON \
      -DERF_USE_CPP:BOOL=ON \
      ..

While performing a cmake -LAH .. command will give descriptions of every option for the CMake project. Descriptions of particular options regarding the testing suite are listed below:

ERF_ENABLE_FCOMPARE – builds the fcompare utility from AMReX as well as the executable(s), to allow for testing differences between plot files

ERF_ENABLE_TESTS – enables the base level regression test suite that will check whether each test will run its executable to completion successfully

ERF_ENABLE_UNIT_TESTS – enables the gtest-based unit-test executable and registers CTest tests with the unit label; this defaults to ON when ERF_ENABLE_TESTS=ON and ERF_ENABLE_REGRESSION_TESTS_ONLY=OFF, and can also be enabled explicitly for unit-only builds

ERF_ENABLE_REGRESSION_TESTS_ONLY – when ERF_ENABLE_TESTS=ON, suppresses unit-test build and registration so only the regression suite is configured

Building the Tests

Once the user has performed the CMake configure step, the make command will build the ERF executable(s) required for each test (for example, the shared erf_exec binary for regression tests and erf_unit_tests for gtest-based unit tests). In this step, it is highly beneficial for the user to use the -j option for make to build source files in parallel.

Running the Tests

Once the test executables are built, CTest also creates working directories for each test within the build directory where plot files will be output, etc. This directory is analogous to the source location of the tests in Tests/test_files.

Where is the executable? With the CMake workflow, the shared regression executable is built under the build tree in Exec (for example, build/Exec/erf_exec), and all regression/canonical test input decks are run using that binary. The gtest unit-test executable is built as build/Tests/Unit/erf_unit_tests. When enabling unit tests locally, make sure the GoogleTest submodule is initialized first, for example with git submodule update --init Submodules/googletest (or git submodule update --init --recursive after cloning). ERF unit tests use a custom GoogleTest main() that initializes and finalizes AMReX before and after RUN_ALL_TESTS() so the same harness can be used for CPU and GPU-oriented tests.

To run the test suite, run ctest in the build directory. CTest will run the tests and report their exit status. Useful options for CTest are -VV which runs in a verbose mode where the output of each test can be seen. -R where a regex string can be used to run specific sets of tests. -j where CTest will bin pack and run tests in parallel based on how many processes each test is specified to use and fit them into the amount of cores available on the machine. -L where the subset of tests containing a particular label will be run. ERF currently uses ctest -L unit for gtest-based unit tests and ctest -L regression for regression tests. Some GPU workflows currently build the unit-test executable without running it, but the unit-test scaffold is enabled across CPU, CUDA, HIP, and SYCL builds. Output for the last set of tests run is available in the build directory in Tests/Temporary/LastTest.log.

Adding Tests

Developers are encouraged to add tests to ERF and in this section we describe how the tests are organized in the CTest framework. The locations (relative to the ERF code base) of the tests are in Tests. To add a test, first create a problem directory with a name in Exec/RegTests/<prob_name> (for problems to be used as regression tests), Exec/CanonicalTests/<prob_name> (for canonical test cases), or ERF/.Exec_dev/<prob_name> (for problems testing features under development), depending on which type of test is being added. Prepare a suitable input file. As an example, the TaylorGreenVortex problem with input file Exec/RegTests/TaylorGreenVortex/inputs_ex solves a simple advection-diffusion problem. The corresponding regression tests are driven by the input files Tests/test_files/TaylorGreenAdvecting/TaylorGreenAdvecting.i and Tests/test_files/TaylorGreenAdvectingDiffusing/TaylorGreenAdvectingDiffusing.i.

Any file in the test directory will be copied during CMake configure to the test’s working directory. The input files meant for regression test run only until a few time steps. The reference solution that the regression test will refer to should be placed in Tests/ERFGoldFiles/<test_name>. Next, edit the Tests/CTestList.cmake file, add the problem and the corresponding tests to the list. Note that there are different categories of tests and if your test falls outside of these categories, a new function to add the test will need to be created. After these steps, your test will be automatically added to the test suite database when doing the CMake configure with the testing suite enabled.