Debugging tools for HPC

Overview

Debugging tools for high performance computing must cope with large process counts, multiple memory spaces, accelerators, and job schedulers. The aim of this chapter is not to turn you into an expert on every tool, but to give you a practical map of the most common categories of tools, how they are typically used on clusters, and what to expect when you first launch them on parallel jobs.

You have already seen typical bugs in parallel programs and general debugging strategies. Here the focus is on the concrete tools that support those strategies in an HPC environment.

Categories of Debugging Tools in HPC

In HPC, debuggers can be roughly grouped into four practical categories.

First, there are traditional source level debuggers that you may already know from single process development. These are extended or wrapped to support MPI, OpenMP, and hybrid applications.

Second, there are parallel debuggers that can attach to or launch very large MPI or hybrid jobs and present aggregated views of thousands of processes.

Third, there are correctness and error checking tools, which do not behave like classic interactive debuggers, but instead run your job with extra checks that detect memory errors, race conditions, deadlocks, and other correctness issues.

Fourth, there are runtime trace and log based tools, which record what happens during a parallel run and allow you to inspect it after the fact. These are often introduced for performance analysis, but many also help debug tricky behavior such as intermittent hangs or wrong message ordering.

Each category has its place. In practice, you will often combine them. For instance, you might use a memory checker to find an invalid access, then reproduce the issue in a small run under an interactive debugger.

Using Classic CLI Debuggers in an HPC Context

Most HPC systems provide familiar command line debuggers such as gdb for C and C++, or lldb on LLVM based systems. These tools work on a single process or a small number of processes, so their role in HPC debugging is to narrow down problems and confirm hypotheses on smaller test cases.

On a cluster, you almost never run gdb directly on the login node. Instead, you launch it inside a job allocation. With SLURM, that often means using srun with the --pty option to request an interactive shell on a compute node. Inside that shell you can start gdb with your executable and run the program with reduced input sizes or a minimal number of MPI ranks or threads.

For MPI applications, there is an important distinction between attaching a debugger to one process and debugging all ranks. On many systems the MPI implementation integrates with gdb through helper wrappers, so you can start something like srun -n 4 gdb ./a.out and get one debugger instance per rank. This can become unwieldy once the number of ranks grows. It is common to start by debugging only rank 0 or a small subset of ranks to observe the general behavior, then move to more specialized tools once you suspect cross rank interactions.

For threaded programs that use OpenMP or similar models, gdb can show different threads in a single process. The debugger can switch between threads, show their call stacks, and allow you to step through code to see how variables change. In an HPC setting this must again be performed on a compute node, and it often uses a build compiled with debug symbols and with optimizations reduced.

Parallel Debuggers for MPI and Hybrid Codes

Parallel debuggers are designed specifically for large scale MPI applications and hybrid MPI plus thread based codes. They provide a single user interface that controls many processes across multiple nodes. This is essential once you move beyond a handful of ranks and need to reason about collective operations, synchronization across ranks, or rank specific failures.

Common commercial and academic tools in this category include DDT (which is now part of the Arm Forge suite), TotalView, and some vendor specific tools integrated with MPI stacks. Although their user interfaces differ, they share key capabilities that matter to you as an HPC beginner.

They can launch an MPI job through the scheduler on your behalf or attach to a running job. This is typically done with a front end that runs on the login node and back end agents that run on the compute nodes. As a user you specify the MPI launcher command, the number of ranks, and any job script parameters, and the debugger does the orchestration.

They provide aggregated views of variables and call stacks. Instead of stepping one rank at a time, you can pause all ranks, look at where each rank is in the code, and detect patterns such as half of the ranks waiting in a MPI_Recv while others are still performing computation. You can inspect a variable across all ranks, see which ranks have a different value, and filter to just those that differ.

Many parallel debuggers support group operations. You can apply breakpoints to all ranks or to a subset, such as only a particular communicator or only ranks on a certain node. This lets you reproduce bugs that only appear for a subset of your processes. Tools also allow conditional breakpoints on rank id or variable values, which is particularly useful for debugging logic that behaves differently across ranks or for specific data partitions.

Hybrid debugging means the tool understands both processes and threads. You can see all threads per rank, examine synchronization primitives, and reason about ordering between MPI calls and thread activity. This combination is important in modern node architectures, where each MPI rank may run with many threads.

Finally, parallel debuggers usually offer deadlock analysis features. When you suspect a hang, you can pause the job and the debugger can classify ranks based on whether they are stuck in blocking communication or are still active. This quickly reveals mismatched collectives or inconsistent send and receive patterns across ranks.

Memory Debugging and Correctness Checking Tools

Many hard to find bugs in HPC codes come from memory misuse or subtle correctness violations that a normal debugger does not automatically expose. Correctness checking tools instrument your program so that it observes its own behavior during execution, usually at significant runtime cost. In exchange you get early detection of defects that could otherwise lead to non reproducible crashes, silent data corruption, or hangs.

On CPU based codes, the most widely known tool family is Valgrind. It provides components such as Memcheck for detecting use of uninitialized memory, invalid reads and writes, and memory leaks. Using Valgrind on an HPC cluster follows the same pattern as an interactive debugger. You request a node through the scheduler, run your program under Valgrind with a small problem size and reduced number of ranks or threads, then inspect the detailed reports. Many MPI stacks work correctly under Valgrind, but the overhead is high, so you usually start with a very small run.

Thread correctness tools such as ThreadSanitizer and similar sanitizers integrated into modern compilers can detect data races and incorrect synchronization in multithreaded codes. They are enabled through compiler flags and linked runtimes. In an HPC environment these builds tend to be used on small tests, because the overhead and memory usage are substantial. The same applies to AddressSanitizer, which detects buffer overflows and use after free bugs, and UndefinedBehaviorSanitizer, which detects operations that rely on undefined behavior in the language standard.

MPI correctness tools form another important group. They check for common mistakes such as mismatched message sizes, illegal communicator usage, or inconsistent collective calls across ranks. Examples include tools that intercept MPI calls at runtime to detect patterns that would lead to deadlocks or non portable behavior. Some tool suites that are primarily performance oriented, such as certain MPI trace analyzers, also incorporate correctness checks about ordering and matching of messages.

There are also specialized tools for checking floating point behavior, array bounds, and domain specific constraints. Some numerical libraries and application frameworks ship with their own internal assertions and consistency check modes. In HPC debugging practice, it is common to first enable these internal checks, run with small test cases, and only reach for more invasive third party tools when a problem persists.

Debugging Tools for GPU and Accelerator Codes

When your code uses GPUs or other accelerators, you must debug both host code and device code. Each major GPU vendor provides its own suite of debugging and analysis tools that understand the corresponding programming models such as CUDA and OpenACC.

GPU debuggers can set breakpoints in kernels running on the device, inspect variables in device memory, and step through device code similarly to how you do it on the CPU. In an HPC setting these tools must integrate with batch scheduling and often require that the debugger front end runs on a node with graphic capabilities or that you connect from your local machine to a cluster front end over a remote desktop or tunneling session.

Most GPU debugging tools are tightly coupled to specific toolchains. For example, CUDA oriented debuggers are aware of kernel launches, streams, and asynchronous memory transfers, and they can correlate device side call stacks with the corresponding host side context. OpenACC aware tools can display mappings between host arrays and device memory regions and report on whether data is present or needs to be transferred.

From a practical perspective, GPU debugging on a cluster usually involves requesting exclusive access to a GPU node, because you need to ensure that the debugger can control the device without interference from other users' jobs. This must be reflected in the job script through appropriate resource requests.

Correctness tools exist for GPUs as well. These tools can detect data races in kernels, shared memory misuse, out of bounds accesses, and memory initialization errors on the device. Just like their CPU counterparts, they introduce substantial overhead and are normally used on small test problems. They complement host based sanitizers by covering the accelerator side of the application.

Scheduler Integration and Remote UI Considerations

Interactive debugging on a parallel cluster always occurs under the control of the job scheduler. You do not circumvent the scheduler to run debuggers. Instead, debuggers either launch the job through the scheduler or attach to a job that was launched with debugging in mind.

For launch based workflows, many parallel debuggers and correctness tools provide templates for SLURM or other batch systems. You configure the number of nodes, ranks, and threads, and the tool constructs an appropriate job script or submission command. The debugger's back end processes then run as part of the allocated job and communicate back to the front end over the network. From your perspective this hides the details of distributed launching, but it is still subject to queue policies and resource limits.

Attach based workflows start with a normal job submission. You include options that prevent the job from progressing too far before you attach, such as waiting for user input or sleeping in an early phase. Once the job is running on compute nodes, you use the debugger front end to locate the job and connect to its processes. This method is useful for reproducing bugs that appear only in the normal execution environment with realistic job scripts.

On many HPC systems the debugger graphical interface does not run directly on the compute nodes. Instead, it runs on a login node or on your local workstation and connects to back end agents started in the job allocation. To use a graphical interface securely across the network, clusters often require SSH tunneling or specific remote desktop solutions. Text based interfaces, including the command line mode of many tools, avoid graphical issues and are often easier to integrate with batch systems.

Because cluster policies restrict long interactive sessions, you must be aware of time limits on interactive allocations. This affects how you plan debugging sessions. It is common to start with short allocations and small problem sizes and extend only when necessary. For large jobs, you may rely more on postmortem debugging, examining core files, logs, and traces from failed runs, rather than on live interaction.

Choosing the Right Tool for the Situation

With many categories of tools available, it is important to match the tool to the type of bug and the scale of the problem.

For consistent crashes in a single rank or thread, start with a traditional debugger like gdb on a reduced input. Combine it with address and undefined behavior sanitizers built into the compiler to quickly catch memory violations and language level errors.

For suspected race conditions or memory leaks in threaded codes, use thread and memory sanitizers or tools like Valgrind on very small test cases. Narrow down the offending region and only then switch to interactive debugging if needed.

For MPI level issues such as hangs in collectives, mismatched sends and receives, or rank dependent behavior, employ a parallel debugger on a job with a moderate number of ranks. Use the debugger's aggregated communication views and deadlock detection to understand which ranks disagree about the communication pattern.

For problems that involve both CPU and GPU, check vendor recommended GPU debuggers and device side checkers. Focus first on verifying correct data movement and kernel launches, then debug kernel logic if incorrect results persist.

For elusive bugs that appear only at large scale, rely on traces and logs, possibly with lightweight correctness checks enabled. Analyse patterns offline to generate hypotheses. Then reproduce the behavior at smaller scale under interactive debuggers and correctness tools.

Finally, remember that building your code with debug symbols and without aggressive optimization is essential for effective source level debugging. Release builds are optimized for performance, but they often obscure line level behavior and may inline or reorder code so that stepping through it becomes confusing.