Introduction to CMake

Basic idea and role of CMake in HPC

CMake is a build system generator. It does not compile your code directly. Instead, it reads configuration files that you provide, then generates native build files such as Unix Makefiles or project files for IDEs. On most HPC systems, you will combine CMake with compilers and libraries that are already provided by the cluster environment.

The main reason CMake matters in HPC is portability. A single CMake-based project can be configured on a laptop, a Linux workstation, and multiple clusters, each with different compilers, MPI stacks, and math libraries, without manually editing platform specific build files. You describe what you want to build and which libraries you need. CMake figures out how to translate that into the correct compiler and linker commands on each system.

Typical workflow with CMake

A CMake driven workflow usually has three steps: configure, build, and optionally install.

First, you create a source tree that contains your C and C++ and Fortran files together with one or more CMakeLists.txt files. These files describe the project, its targets, and dependencies.

Second, you run cmake to configure the build in a separate directory. This is called an out of source build and keeps generated files out of your source tree. A typical sequence on a cluster is:

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j4

The command cmake .. tells CMake to read the CMakeLists.txt files from the source directory above the build directory. The CMake configuration step detects compilers, checks for libraries, and generates a Makefile or other build files. After that, make actually compiles the code.

If your project supports installation, you can run:

make install

optionally with DESTDIR or custom install prefixes if you do not have write access to system directories, which is common on HPC clusters.

Basic structure of a CMake project

A minimal CMake project has a top level CMakeLists.txt that sets the project name and language, then defines one or more targets.

A very small example for a C program:

cmake_minimum_required(VERSION 3.18)
project(my_hpc_code LANGUAGES C)
add_executable(my_hpc_app main.c)

The command cmake_minimum_required declares the oldest CMake version that is supported. The command project gives a name to the project and declares which languages are in use. The command add_executable defines a binary target named my_hpc_app that is built from main.c.

For a library, you would use:

add_library(my_hpc_lib matrix.c solver.c)

Targets are central in modern CMake. Each target corresponds to something that can be built such as an executable or a library, and you attach include directories, compiler features, and linked libraries to targets.

Out of source builds and build directories

On HPC systems, you often build the same code with different configurations, for example a debug build, an optimized build, and perhaps a build with GPU support. Out of source builds make this manageable.

Suppose your source directory is ~/code/my_hpc_project. You can create separate build directories:

cd ~/code/my_hpc_project
mkdir build-debug build-release
cd build-debug
cmake .. -DCMAKE_BUILD_TYPE=Debug
make -j8
cd ../build-release
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j8

Your source files remain untouched while each build directory has its own CMakeCache.txt, generated Makefile, and compiled object files. You can remove a build directory safely without affecting the source.

On clusters, you might keep build directories on a fast filesystem such as a node local SSD or a scratch area. The CMake files themselves stay in your home or project space but the heavy build artifacts can go to a performance oriented location.

Selecting compilers and build types on clusters

CMake uses default compilers from your environment unless you tell it otherwise. On HPC systems, the environment is often prepared through modules. You load the desired compiler and MPI modules, then run cmake.

For example:

module load gcc/12.1
module load openmpi/4.1
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j16

The option -DCMAKE_BUILD_TYPE=Release asks CMake to use the Release configuration, which usually enables optimization and disables debug symbols. Common single configuration build types are:

Debug
Release
RelWithDebInfo
MinSizeRel

Single configuration generators, such as Unix Makefiles, use CMAKE_BUILD_TYPE to decide which flags to use. On most clusters this is what you will use. Multi-configuration generators such as Visual Studio support several configurations in one build tree, but are less relevant for typical HPC Linux environments.

If you must select nondefault compilers, you can use environment variables during the first configuration:

CC=icc CXX=icpc FC=ifort cmake .. -DCMAKE_BUILD_TYPE=Release

or you can pass them as CMake variables:

cmake .. -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc -DCMAKE_Fortran_COMPILER=ifort

These choices are cached in CMakeCache.txt. If you want to change compilers after the first configuration, it is safer to delete the build directory and start again.

Controlling compiler flags with CMake

CMake provides higher level interfaces to set compiler options. For quick experiments you can use variables like CMAKE_C_FLAGS or CMAKE_CXX_FLAGS, but the recommended method is to set options on targets.

An example with target properties:

add_executable(my_hpc_app main.c solver.c)
target_compile_options(my_hpc_app PRIVATE -Wall -Wextra)
target_compile_definitions(my_hpc_app PRIVATE USE_FAST_MATH)
target_include_directories(my_hpc_app PRIVATE ${CMAKE_SOURCE_DIR}/include)

This approach keeps flags and include paths attached to the code they affect. It is easier to understand and more robust with respect to cross platform differences. You can also use generator expressions and conditions so that certain flags apply only to specific compilers or build types.

For build types, you can define options depending on CMAKE_BUILD_TYPE:

if(CMAKE_BUILD_TYPE STREQUAL "Debug")
    target_compile_options(my_hpc_app PRIVATE -O0 -g)
elseif(CMAKE_BUILD_TYPE STREQUAL "Release")
    target_compile_options(my_hpc_app PRIVATE -O3)
endif()

On HPC clusters, a typical pattern is to select high optimization levels and architecture specific flags in Release builds, while keeping Debug builds simple and easy to debug.

Linking with libraries used in HPC

CMake has a standard way to find and use external libraries. For system or package managed libraries, you can use find_package. For example, to use an installed FFTW library:

find_package(FFTW3 REQUIRED)
add_executable(my_fft_app main.c)
target_link_libraries(my_fft_app PRIVATE FFTW3::FFTW3)

The find_package command locates headers and libraries and creates imported targets such as FFTW3::FFTW3. Linking against these targets is preferable to specifying raw paths. For some HPC libraries there are provided CMake config files, while others are found through Find*.cmake modules that might be bundled with CMake or supplied by your project.

When using math libraries that depend on ordering or additional system libraries, CMake helps keep these details encapsulated. For example, linking my_hpc_app against LAPACK and BLAS can be as simple as:

find_package(BLAS REQUIRED)
find_package(LAPACK REQUIRED)
add_executable(my_solver main.c)
target_link_libraries(my_solver PRIVATE LAPACK::LAPACK BLAS::BLAS)

The actual linker line might include many library names and paths, but CMake hides this complexity behind the imported targets.

CMake and MPI based HPC codes

MPI codes built with CMake use the same target based approach. If your cluster provides an MPI aware C compiler wrapper such as mpicc, you can allow the compiler environment to handle MPI, or you can explicitly use CMake’s MPI support.

A basic example using find_package:

find_package(MPI REQUIRED)
add_executable(my_mpi_app main.c)
target_link_libraries(my_mpi_app PRIVATE MPI::MPI_C)

The imported target MPI::MPI_C includes the proper include directories, compiler definitions, and libraries for MPI. For C++, you would use MPI::MPI_CXX, and for Fortran MPI::MPI_Fortran, depending on your CMake version and MPI configuration.

On HPC clusters, the correct MPI implementation is typically selected by loading the corresponding module before configuring the project. CMake then finds and uses that particular MPI installation.

Using CMake with GPU and accelerator builds

For accelerator enabled codes, you often must add device specific compiler flags and link against GPU runtime libraries. CMake offers language support for CUDA and can manage OpenACC and OpenMP offload partially through compiler options.

A minimal CUDA example:

cmake_minimum_required(VERSION 3.21)
project(gpu_example LANGUAGES CXX CUDA)
add_executable(gpu_app main.cu)
set_target_properties(gpu_app PROPERTIES CUDA_SEPARABLE_COMPILATION ON)

You can control GPU architectures through target options, for example:

target_compile_options(gpu_app PRIVATE $<$<COMPILE_LANGUAGE:CUDA>:--generate-code=arch=compute_80,code=sm_80>)

On clusters with vendor specific compilers, you might represent GPU related options with target_compile_options and target_link_libraries and keep the CMake logic conditional on the compiler or on CMake options.

For portability across different clusters, you can define configurable cache variables such as GPU_ARCH and let users specify them at configuration time:

option(ENABLE_GPU "Build with GPU support" ON)
set(GPU_ARCH "sm_80" CACHE STRING "GPU architecture")
if(ENABLE_GPU)
    target_compile_options(gpu_app PRIVATE
        $<$<COMPILE_LANGUAGE:CUDA>:--generate-code=arch=compute_80,code=${GPU_ARCH}>)
endif()

Users can then run:

cmake .. -DENABLE_GPU=ON -DGPU_ARCH=sm_70

Handling include directories and source organization

CMake encourages a structured layout that is helpful on larger HPC projects. In many cases, you will split your code into subdirectories, each with its own CMakeLists.txt. The main file then includes these with add_subdirectory. Inside each subdirectory you define libraries that represent logical components of your application and then link them together.

An example structure:

src/core/CMakeLists.txt defines core library.
src/solver/CMakeLists.txt defines solver library.
apps/CMakeLists.txt defines executables using these libraries.

Then in the top level CMakeLists.txt:

add_subdirectory(src/core)
add_subdirectory(src/solver)
add_subdirectory(apps)

Include directories can be attached to each library target with target_include_directories. Other targets that link to the library inherit those include directories when the scope is PUBLIC or INTERFACE.

For example:

add_library(core matrix.c vector.c)
target_include_directories(core
    PUBLIC
        ${CMAKE_SOURCE_DIR}/include
)
add_executable(my_app main.c)
target_link_libraries(my_app PRIVATE core)

The executable my_app automatically receives include access to ${CMAKE_SOURCE_DIR}/include through the link to core.

Configuring and using options in CMake

For HPC applications, you often want to enable or disable features such as MPI, OpenMP, GPU support, or specific numerical kernels. CMake provides the option command to define toggles that show up in the cache.

Example:

option(ENABLE_OPENMP "Enable OpenMP parallelism" ON)
add_executable(my_app main.c)
if(ENABLE_OPENMP)
    find_package(OpenMP REQUIRED)
    target_link_libraries(my_app PRIVATE OpenMP::OpenMP_C)
endif()

Users can control these at configure time:

cmake .. -DENABLE_OPENMP=OFF

This pattern integrates nicely with cluster workflows where different builds might be scheduled with or without accelerators or threading.

You can also define numeric or string cache variables with set(... CACHE ...). For example:

set(MAX_GRID_SIZE 1024 CACHE STRING "Maximum grid size for problem setup")
target_compile_definitions(my_app PRIVATE MAX_GRID_SIZE=${MAX_GRID_SIZE})

Users adjust MAX_GRID_SIZE during configuration without editing code, which is useful for performance experiments and scaling studies.

Installing and using CMake based software on clusters

CMake can install executables, libraries, headers, and configuration files into a chosen prefix. On systems without root access, you normally use a personal prefix under your home directory or a shared project space.

A basic installation fragment:

add_executable(my_app main.c)
install(TARGETS my_app
        RUNTIME DESTINATION bin)
install(DIRECTORY include/
        DESTINATION include)

Then you can configure the project with:

cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/local/my_hpc_project
make -j8
make install

The binary will be installed in $HOME/local/my_hpc_project/bin and headers in $HOME/local/my_hpc_project/include.

For reusable libraries, you can also export CMake package configuration files, so that other CMake projects can consume your library with find_package. This is common in larger HPC software stacks, although the detailed setup belongs to more advanced material.

Practical tips for using CMake on HPC systems

In practice, several habits will make CMake more effective in an HPC context.

First, always separate your build directory from the source, and avoid running cmake from the source tree. Second, always load your environment modules and set CMAKE_BUILD_TYPE before the first configuration. Third, capture configuration and build commands in scripts so that you can reproduce builds. For example, keep a simple shell script that runs cmake with all your chosen options then calls make. This is consistent with reproducibility goals and simplifies performance comparisons between different configurations.

Finally, remember that CMake caches many configuration variables. If you change compilers, core libraries, or important environment settings, it is often safer to delete the build directory and start again than to try to fix everything in place.

With these basics, you can use CMake as a consistent interface to build, configure, and manage HPC applications across laptops, workstations, and large clusters.