Table of Contents
Layered view of software on HPC systems
When people talk about a “software stack” in HPC, they mean all the software layers that sit between your source code and the hardware. For reproducibility, it matters which layers you use and how they interact.
At the bottom you have the operating system and kernel that are fixed by the cluster administrators. On top of that come system-level libraries, communication libraries such as MPI, math and domain libraries, compilers, and finally your applications or scripts. You may also have container runtimes and environment modules that provide alternate views of the stack.
A helpful mental model is to picture your application calling numerical libraries, which depend on MPI and system libraries, which in turn depend on the kernel and hardware. When you report or recreate a run, you want to be able to reconstruct a compatible version of each relevant layer, not only your own code.
A software stack is the ordered collection of software layers required to build and run an application, from the operating system and system libraries up to compilers, libraries, tools, and the application itself. For reproducibility, you must be able to identify and control these layers.
Typical components of an HPC software stack
Although every cluster has its own policies, most production systems share similar stack components. Only some of them will be visible to you as a user, but all of them influence results and performance.
At the base is the operating system distribution, usually a Linux variant, plus the kernel. Administrators also install basic system libraries such as the C standard library libc, system libm, and low level tools. These are not usually changed by users, but they can differ between clusters or between generations of the same cluster.
Above that you typically find one or more families of compilers, such as GCC, Clang, or vendor compilers. Each compiler version can generate very different machine code, and can interact in subtle ways with libraries, especially threaded libraries or vectorized math routines.
Next come parallel communication layers such as MPI or SHMEM implementations, and vendor drivers and run time libraries for accelerators such as CUDA and ROCm. These pieces control how processes and threads communicate and how data moves between host and device memory.
On top of these lie math and domain libraries, for example BLAS, LAPACK, FFT libraries, solvers, and domain specific toolkits. Many of these libraries are provided in multiple flavors configured for different MPI stacks, compilers, and hardware. Using a version that mismatches your compiler or MPI implementation is a common source of obscure failures.
At the top you have applications, frameworks, and scripting environments such as Python stacks or R distributions. These can in turn depend on lower layers through compiled extensions, numerical backends, and bindings to MPI or GPU libraries.
The key property of a stack is that each layer is usually built against specific versions of the layers below. Modifying one layer without adjusting the others can break the whole stack.
Centrally managed vs user-level stacks
On most clusters the primary software stack is centrally managed by system administrators. They build and install compilers, MPI implementations, numerical libraries, common tools, and possibly complete application suites. This central stack is typically tested for correctness and performance on the specific hardware and interconnect.
A centrally managed stack has important advantages for reproducibility on the same system. You can refer to standardized module names and versions, configuration documentation, and known combinations that are guaranteed to work. When several users rely on the same modules and toolchains, reproducing each other’s environments becomes much easier.
At the same time, many workflows require software that is not provided centrally or require different versions. In these cases users build their own stacks in their home or project spaces. User-level stacks can range from a single custom binary compiled with a central compiler to a full tree of compilers, MPI, libraries, Python environments, and their dependencies.
For reproducibility, user-level stacks increase your responsibility. You must track how you built each component, which dependency versions you used, and where the stack is stored. Without this tracking, it can be very difficult to reconstruct results months later or to move a workflow to another cluster.
Because of this, many sites encourage a hybrid strategy: rely on the central stack for low level components such as compilers, MPI, and core math libraries, and layer your custom applications, Python environments, and domain libraries on top. This reduces the number of components you must manage while still giving flexibility.
Toolchains and compatibility constraints
HPC software stacks are not arbitrary collections of tools. They are usually organized around “toolchains.” A toolchain is a consistent set of compiler, MPI implementation, and often math libraries that are designed and tested to work together. For example, a cluster might define a gcc/12 + openmpi/5 + openblas toolchain, or an intel-oneapi toolchain that bundles compiler, MPI, and math kernels.
The most important compatibility constraints in a stack arise from these toolchains. Libraries that use MPI symbols must match the MPI implementation and often its version. Libraries that expose C++ interfaces must be compatible with the C++ ABI of the compiler that will link the final application. GPU libraries must match driver versions and supported hardware.
If you mix toolchains carelessly, you may encounter runtime crashes, linking errors, or silent misbehavior. For instance, mixing objects compiled with different C++ standard libraries can lead to obscure symbol errors. Linking an application built with one MPI implementation against a library that was built with a different MPI can lead to deadlocks or data corruption.
To avoid this, you should treat the toolchain as the foundation of your stack. Choose a compiler and MPI pair first, then select or build all other compiled libraries against that combination. If you use centrally provided modules, administrators will usually name modules so that their dependencies are clear. If you build your own, you must document which toolchain you used for each component.
For a reproducible software stack, you must fix a consistent toolchain, including compiler, MPI implementation, and key libraries, and ensure that every compiled component in your application is built against this same toolchain.
Versioning and naming conventions
A central challenge for reproducibility is dealing with version numbers across the stack. Each layer has its own versioning scheme and release cycle, so a particular environment is defined not by a single version but by a vector of versions.
On clusters that use environment modules, stack components often follow a naming convention that encodes dependencies, such as hdf5/1.14.0-gcc-12.2-openmpi-5.0 or petsc/3.20-intel-oneapi-2024. This name indicates that the library version and its build were tied to a specific toolchain. When you write down the modules you have loaded, you implicitly capture a large part of the stack.
At the user level, versioning can be more informal. Python virtual environments, for example, track package versions in a requirements file. Build systems such as CMake or autotools may capture versions of dependencies at configure time in cache or build logs. Package managers such as Spack or Conda maintain explicit records of build recipes and dependency trees.
For reproducibility you need to move from implicit to explicit recording. Instead of relying on “latest GCC” or “default MPI,” record exact version identifiers. A complete reproduction may require the kernel version and microarchitecture, but often you can get quite far by fixing compiler, MPI, core numerical libraries, and high level libraries and frameworks.
Software stacks and environment modules
Environment modules are a central mechanism for presenting and composing software stacks on shared systems. While environment modules are covered in more detail elsewhere, it is useful here to see how they shape stacks.
Modules allow administrators to maintain multiple complete or partial stacks side by side. For example, a site may provide an “Intel stack” based on one compiler family, and a “GNU stack” based on another, with several versions of each. The module system encodes which combinations are supported. Loading a base module for a toolchain automatically adjusts your environment variables so that compilers, MPI, and core libraries work together.
Higher level modules, such as numerical libraries and applications, typically depend on a specific toolchain module. The module system can enforce that you only load compatible pieces. This turns your chosen set of modules into an explicit description of your software stack.
From a reproducibility perspective, the module list you have loaded at run time is one of the most important artifacts. It is common practice to print or record the output of module list or an equivalent command at the start of a job. When someone else wants to rerun your job on the same or a similar cluster, the module list serves as a recipe for reconstructing the stack.
Packaging systems for building stacks
Beyond modules, several packaging systems are designed specifically to build and manage HPC software stacks in a reproducible way. These tools encapsulate build instructions, dependency graphs, and environment information, which helps to rebuild the same stack later or on another system.
Tools such as Spack or EasyBuild (described elsewhere) treat each software installation as an instance identified by its version, compiler, MPI, and build options. When you request a package, the tool resolves all dependencies and can generate a consistent tree that forms your stack. Because build recipes are explicit, you can record exactly how each component was compiled.
General purpose package managers such as Conda or system package managers are also sometimes used to manage user level stacks, especially for interpreted languages and high level tools. These tools track dependencies and can often export manifest files that list all package versions, which is valuable for reproducibility.
In all these systems, the idea is similar. The software stack is not a random ad hoc combination of builds. It is the result of a declarative description of what you want and how components should be built. That description, once saved under version control or attached to a publication, becomes the primary artifact for re-creating the environment.
Containers as encapsulated software stacks
Containers in HPC, such as those built for Singularity or Apptainer, can be thought of as portable, encapsulated software stacks that include most layers above the kernel. A container image typically contains user space libraries, compilers or run times, language environments, and your applications, all frozen at specific versions.
From the perspective of software stacks, containers shift responsibility for stack construction from the cluster to the user or to an image maintainer. The host cluster still supplies the kernel and hardware, and often the MPI and GPU drivers, but many other parts of the stack are defined inside the container. This can greatly improve reproducibility across systems, because you can run the same image on different clusters.
However, containers do not remove stacking concerns. You still need a clear design for how the container stack interacts with host MPI, drivers, and performance libraries. In many HPC workflows, the container is layered on top of a host toolchain, such as “MPI on the host, application in the container.” Understanding which parts of the stack live inside or outside the container is essential to achieve both correctness and high performance.
Software stacks and numerical reproducibility
Software stacks influence not only whether your code runs, but also the exact numerical results you obtain. Different compilers and math libraries may apply different optimizations or algorithms that produce slightly different floating point behavior. GPU backends and threaded BLAS implementations can change operation ordering and therefore rounding errors.
On some systems you may have the option to select between “fast” and “reproducible” modes in numerical libraries. These modes may control the use of fused operations, parallel reductions, or algorithmic shortcuts. These choices are part of your software stack configuration, even if they are controlled by environment variables or runtime flags rather than distinct versions.
To manage this, you should treat algorithmic settings, library backends, and tunable performance options as part of the stack description. When you report results, it is helpful to note not only which library you used, but also any significant environment variables, configuration files, or run time flags that affect numerical behavior.
Strategies for documenting and preserving stacks
For reproducibility, it is not enough to construct a suitable stack. You must also be able to reconstruct it later. This requires a systematic strategy for documenting and preserving the relevant information.
On clusters that use modules, you can start by capturing the full module environment. At the beginning of a job script, you might record the output of a command that lists loaded modules into a log file. For user level package managers, you can export a lockfile or environment specification. Build systems often generate configuration logs that should be kept alongside your source code and input data.
It is common to record the compiler version, MPI implementation, and a short summary of important libraries directly in the output of a job. Including version strings at program startup, obtained from library APIs or compile-time macros, makes it easier to reconstruct a stack from logs alone.
For long running projects, freezing a specific combination of modules and package versions and giving it a label, such as a tag in version control or an internal “environment version,” can help coordinate teams. Everyone on the project can then refer to “Environment v3” and know which stack that implies.
Finally, if you build custom stacks with a packaging system or containers, the build recipes, Dockerfiles, or package specifications should be treated as first class project artifacts under version control. The image files themselves are useful, but the recipes are what make the stack reproducible in the long term, especially after systems are upgraded.
A reproducible workflow must record enough information about the software stack, including toolchain, libraries, configuration, and environment settings, so that the same stack can be reconstructed later on the same or a comparable system.
Evolution of software stacks over time
Clusters do not stand still. Operating systems are updated, compilers gain new versions, MPI implementations evolve, and libraries change defaults. Over the lifetime of a project, your software stack will inevitably face upgrades or deprecations.
One challenge is to balance the need for stability to preserve reproducibility with the need to adopt new features, performance improvements, and security fixes. Many sites handle this by providing multiple generations of stacks in parallel. An older “frozen” stack remains available for legacy workflows, while a newer “current” stack receives updates. Over time, frozen stacks may be retired, which makes your documentation even more valuable.
From the user side, you can plan for this evolution by building test suites and validation runs that you can use when moving from one stack to another. If you can show that results under the new stack agree with a baseline within acceptable tolerances, you can adopt the updated stack without losing confidence in your workflows.
Understanding and managing software stacks in this way connects directly to reproducibility. Instead of treating the environment as a fragile accident, you treat it as a planned, documented, and testable component of your scientific or engineering process.