Debug builds

What is a Debug Build?

A debug build is a version of your program compiled specifically to make finding and fixing bugs easier, not to run as fast as possible. In HPC, you typically:

• Develop and test with debug builds.
• Benchmark and run production jobs with optimized (release) builds.

Debug builds trade performance for:

• Better error checking
• Easier debugging with tools
• More informative crashes

Typical Characteristics of Debug Builds

While details vary by compiler, most debug builds have these properties:

• No or low optimization
• Instructions stay close to your source code.
• Variables are not aggressively reordered or removed.
• Stepping through code in a debugger is intuitive.
• Full debug information
• Symbols (function names, variable names) are preserved.
• Line-number information is embedded so tools can map addresses back to source files.
• Extra runtime checks (if enabled)
• Array bounds checks (language- or library-dependent).
• More assertions active.
• Optional sanitizers (for C/C++ and Fortran).
• Larger binaries, slower code
• Debug info makes the executable larger.
• Minimal optimization makes code significantly slower than a release build.

Common Compiler Flags for Debug Builds

Exact flags depend on the compiler family. Here are the typical patterns in HPC compilers.

GCC (gcc, g++, gfortran)

Core debug flags:

• -g
  Generate debugging information for tools like gdb and many profilers.
• -O0 or -O1
  Low optimization to keep source and machine code closely aligned. -O0 is often used for deepest debugging; -O1 is a compromise that can make code run less painfully slow while still being debuggable.
• -Wall -Wextra
  Enable a broad set of warnings to catch common mistakes early (technically compile-time, but usually part of a “debug configuration”).

Example debug compile commands:

• C:
  gcc -g -O0 -Wall -Wextra -o myprog_debug main.c
• C++:
  g++ -g -O0 -Wall -Wextra -o myprog_debug main.cpp
• Fortran:
  gfortran -g -O0 -Wall -Wextra -o myprog_debug main.f90

Optional but very useful:

• -fsanitize=address — detect invalid memory accesses (C/C++/Fortran).
• -fsanitize=undefined — detect undefined behavior.
• -fno-omit-frame-pointer — keeps stack frames easier to inspect in debuggers and profilers (sometimes default at -O0).

Intel oneAPI (icx/icpx/ifx, classic icc/ifort)

Core debug flags:

• -g
• -O0 or -O1

Example:

• C/C++:
  icx -g -O0 -debug full -o myprog_debug main.c
• Fortran:
  ifx -g -O0 -debug full -o myprog_debug main.f90

Some Intel-specific helpful flags (availability may vary between classic and oneAPI):

• -traceback — better stack traces on crashes.
• -check all (Fortran) — runtime checks (bounds, uninitialized, etc.).
• -fp-model precise — more predictable floating-point behavior (avoids aggressive reordering).

LLVM/Clang and Flang

Core debug flags are similar to GCC:

• -g
• -O0 or -O1
• -Wall -Wextra (for C/C++)

Examples:

• C++:
  clang++ -g -O0 -Wall -Wextra -o myprog_debug main.cpp
• Fortran (flang, availability depends on system):
  flang -g -O0 -o myprog_debug main.f90

Build-Type Separation: Debug vs Release

In HPC projects, you almost never want to manually toggle flags every time. Instead, you define two (or more) build types:

• Debug: -g -O0 plus checks and warnings.
• Release (optimized): something like -O3 -g or -O3 plus architecture flags and maybe -DNDEBUG to disable assertions.

Key ideas:

• Keep separate build directories: e.g. build-debug/ and build-release/.
• Never mix debug and release object files; always clean or separate builds.

Debug Builds in Make-based Workflows

A typical pattern in a Makefile is to define variables for flags and build types.

Example:

CC      = gcc
CFLAGS_DEBUG   = -g -O0 -Wall -Wextra
CFLAGS_RELEASE = -O3 -march=native -DNDEBUG
# Default build type
CFLAGS = $(CFLAGS_DEBUG)
all: myprog
myprog: main.o solver.o
	$(CC) $(CFLAGS) -o $@ $^
main.o: main.c
	$(CC) $(CFLAGS) -c $<
solver.o: solver.c
	$(CC) $(CFLAGS) -c $<
debug: clean
	$(MAKE) CFLAGS="$(CFLAGS_DEBUG)"
release: clean
	$(MAKE) CFLAGS="$(CFLAGS_RELEASE)"
clean:
	rm -f myprog *.o

Usage:

• Debug build:
  make debug
• Release build:
  make release

This simple pattern keeps debug and release configurations defined in one place and easy to switch.

Debug Builds in CMake-based Workflows

CMake has first-class support for build types. For single-config generators (e.g. Makefiles, Ninja):

• Configure a debug build:

cmake -B build-debug -DCMAKE_BUILD_TYPE=Debug

• Configure a release build:

cmake -B build-release -DCMAKE_BUILD_TYPE=Release

Then:

• Build debug:
  cmake --build build-debug
• Build release:
  cmake --build build-release

CMake sets default flags for each build type using variables like:

• CMAKE_C_FLAGS_DEBUG
• CMAKE_C_FLAGS_RELEASE
• CMAKE_CXX_FLAGS_DEBUG
• CMAKE_Fortran_FLAGS_DEBUG

Typical defaults (simplified):

• Debug: -g -O0
• Release: -O3 -DNDEBUG

You can append your own flags in CMakeLists.txt:

set(CMAKE_C_FLAGS_DEBUG   "${CMAKE_C_FLAGS_DEBUG} -Wall -Wextra")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Wall -Wextra")

Or use target_compile_options scoped to specific targets and configurations:

target_compile_options(myprog PRIVATE
  $<$<CONFIG:Debug>:-Wall -Wextra>
  $<$<CONFIG:Release>:-DNDEBUG>
)

For multi-config generators (e.g. Visual Studio, some Ninja setups), you choose the configuration at build time:

• cmake --build build --config Debug
• cmake --build build --config Release

Using Debug Builds with HPC Debuggers

The value of a debug build in HPC is largely about how well you can inspect the running program.

With -g and low optimization, you can:

• Set breakpoints by file and line: e.g. break solver.cpp:120.
• Inspect local variables: print x, print my_array[3].
• Step through code line-by-line.

For MPI/OpenMP/OpenACC programs, HPC debuggers (such as gdb for simple cases, or cluster tools like DDT, TotalView, etc.) rely heavily on debug symbols. Without a debug build, they often show:

• Mangled function names only.
• Missing variable names.
• Confusing stepping and backtraces.

On clusters, you usually:

1. Compile your code with debug flags on a login node.
2. Run the parallel program under the debugger on compute nodes (often via a job script or an interactive allocation).
3. Use the debugger’s GUI or CLI to attach to multiple processes/threads.

Debug Builds and Runtime Checks

Debug builds are a good place to turn on extra runtime checking. Examples (compiler-, language-, and library-dependent):

• Assertions
  In C/C++:
• Use assert(condition); from <assert.h>.
• In debug builds, you compile without -DNDEBUG so assertions stay active.
• In release builds, -DNDEBUG can remove them for performance.
• Bounds and error checks in numerical libraries
  Some libraries or frameworks offer configurable checks that slow things down. Enable them for debug builds; disable for release.
• Fortran runtime checks (Intel, GNU)
  Flags like -check all (Intel Fortran) or -fcheck=all (gfortran) can detect:
• Array bounds violations
• Use of uninitialized variables
• Argument mismatches
• Sanitizers
• AddressSanitizer (-fsanitize=address)
• UndefinedBehaviorSanitizer (-fsanitize=undefined)
  These are often enabled only for debug builds because they significantly slow the program and increase memory usage.

In HPC contexts, you may want to run smaller test problems when using heavy checks, to avoid long runtimes and resource waste.

Trade-offs of Debug Builds in HPC

Key trade-offs relative to optimized builds:

• Performance cost
• Programs can run 5–50× slower or more.
• This directly impacts job time and queue usage.
• Resource usage
• Larger memory footprint (debug info, extra checks).
• Binaries can be much larger; this can matter on network filesystems.
• Scheduler impact
• You generally want to avoid very large, long-running debug jobs.
• Instead, debug on:
• Fewer nodes
• Smaller input sizes
• Shorter wall times

Because of this, a common pattern is:

1. Debug logic and correctness on a workstation or single node with debug builds.
2. Spot-check on the cluster with small parallel runs, still debug builds.
3. Switch to an optimized build for scaling tests and production jobs.

Practical Workflow Recommendations

To use debug builds effectively in an HPC environment:

• Always define a clear, repeatable way to build debug versions
• make debug / make release
• or cmake -DCMAKE_BUILD_TYPE=Debug vs Release.
• Use separate directories/names
• myprog_debug vs myprog
• build-debug/ vs build-release/.
• Keep compilation flags under version control
• Store Makefile/CMakeLists.txt so you can reproduce how a debug binary was built when you analyze bugs.
• Log build information
• Optionally embed version info or use a --version option that reports build type, compiler, and flags.
• Match debug builds to the environment
• Build on the same architecture and with the same MPI/OpenMP configuration as the production environment, to reproduce bugs that depend on parallel behavior.
• Start small
• For debug builds with heavy checks, trim down problem size and node count until you locate the issue, then re-check at larger scale only as needed.

Summary

Debug builds in HPC:

• Are compiled with -g and low optimization (e.g. -O0).
• Include rich debug info and often extra runtime checks.
• Are essential for effective use of debuggers and runtime analysis tools.
• Are significantly slower than optimized builds and should usually be run on reduced-size test cases and fewer resources.
• Are managed alongside release builds via build systems (Make, CMake) that provide a clean, reproducible way to switch between configurations.