Software installation concepts

Installing Software in an HPC Context

In a high performance computing environment, installing software is not just about running a package manager as you might do on a personal laptop. Software must work across many nodes, coexist with other users, and integrate with the cluster environment and its filesystems. This chapter introduces the main concepts and workflows you will encounter when installing or requesting software on an HPC system, without going into the details that belong to the chapters on environment modules or full software stacks.

System-wide vs User-level Installation

On an HPC cluster, there is a fundamental distinction between software installed by system administrators and software managed by individual users.

System-wide installations are maintained by the HPC support team. These are placed in shared locations that are visible on all compute nodes, often on a parallel filesystem. Such installations are configured, tested, and documented as part of the cluster software stack. You typically access them through environment modules, which are discussed in another chapter.

User-level installations are created and managed by you, inside your home directory or a project directory. These installations are not visible to other users by default. You use them when the system does not provide the version you need, when you need experimental or very recent software, or when you want to modify and build software yourself.

The choice between system-wide and user-level installation affects where files live, which compilers and libraries are used, and how you set your environment. It also has implications for performance, reproducibility, and support. Many HPC sites encourage users to rely on the centrally provided stack for most work, and to build their own copies only when necessary.

Basic Software Layout: Prefixes and Paths

Most traditional Unix software follows a similar layout, controlled by a prefix directory. The prefix is a directory under which the installation places binaries, headers, libraries, and documentation.

For example, if the prefix is /opt/mysoftware, the installed files usually end up in subdirectories such as:

/opt/mysoftware/bin for executables.
/opt/mysoftware/lib or /opt/mysoftware/lib64 for libraries.
/opt/mysoftware/include for header files.
/opt/mysoftware/share for data and documentation.

For user-level installs, a common pattern is to use a directory such as $HOME/.local, $HOME/software, or a project directory like /path/to/project/soft. Many build systems allow you to set the prefix with an option such as --prefix=/path/to/prefix during configuration.

To use an installed package, your shell environment must know where to find its components. This involves variables such as:

PATH for locating executables under prefix/bin.
LD_LIBRARY_PATH or LD_PRELOAD in some cases, for shared libraries under prefix/lib.
CPATH, LIBRARY_PATH, or language-specific variables for headers and libraries when compiling.

The details of environment management, including modules and shell configuration, are treated elsewhere. The important concept here is that the prefix defines the root of the installation, and environment variables connect that installation to your build and run-time environment.

System Package Managers vs HPC Builds

On typical Linux workstations, you install software with distribution package managers such as apt, dnf, or yum. On shared HPC systems, you usually do not have administrative privileges, so you cannot use these tools to install system-wide packages. Even if you could, distribution packages may be compiled with generic options that do not take advantage of the specific CPUs, interconnects, or libraries available on the cluster.

In HPC, there are three broad ways software tends to be managed.

First, system administrators may still use OS-level package managers internally to build the base environment, but this is invisible to you. You access that software through environment modules or fixed paths.

Second, many sites provide HPC-oriented meta package managers or build systems such as Spack or EasyBuild. These tools can compile multiple versions of software with different compilers and library dependencies, and install them into separate prefixes. They are designed to manage complex dependency trees and to produce reproducible builds.

Third, for user-level installations, you will often compile from source without root privileges, using language specific tools or generic build systems. For example, you might use pip for Python packages, R CMD INSTALL for R packages, conda or mamba for scientific environments, or CMake and Make for compiled code. These tools support user-local prefixes so that everything installs under $HOME or another user-owned directory.

The key concept is that in HPC, software is usually compiled and installed specifically for the cluster hardware and environment, rather than relying on generic binary packages.

Building from Source: Configure, Build, Install

Many HPC applications and libraries are distributed as source code. Installing them typically involves a three stage process.

First, configuration. In a classic Autotools based project, you run a script named configure, often with options such as:

./configure --prefix=$HOME/software/myapp \
            CC=mpicc CXX=mpicxx FC=mpif90

Configuration detects the compilers, checks for required libraries, and prepares makefiles. In CMake based projects, you call cmake with a build directory and the desired options. CMake then generates platform specific build files.

Second, compilation. For Make based builds, you run make or make -j N where N is the number of parallel build jobs. For CMake generated builds, you may call cmake --build . or directly invoke make or another underlying build tool. On an HPC cluster, you must ensure that you are using the correct compiler wrappers and modules for MPI, math libraries, or GPU support. These choices affect performance and compatibility at runtime.

Third, installation. After a successful build, you run make install or the equivalent for your build system. This copies the built artifacts into the chosen prefix. For user installs, no administrative privileges are needed, as long as the prefix is writeable by your account.

The exact commands and options differ between build systems, but the conceptual flow of configure, build, and install remains similar.

Static vs Shared Libraries

Many HPC codes depend on numerical libraries, communication libraries, or I/O libraries. When installing software, it is important to understand the difference between static and shared libraries, because this influences portability, performance, and runtime behavior.

A static library is usually a file with extension .a. When you link an application against a static library, the required object code is copied into the executable at link time. The resulting binary does not need the static library file at runtime. This simplifies deployment but can increase binary size and make updates more cumbersome.

A shared library is a file with extension .so on Linux. When you link against a shared library, the executable records a dependency on that .so file. At runtime, the dynamic linker loads the shared library from the filesystem. This makes it easier to update libraries centrally, and reduces memory duplication when many processes share the same library in memory.

HPC clusters commonly use shared libraries for system-wide installations, because they allow many jobs to share code loaded from parallel filesystems. However, static linking is still used in some contexts, for example for performance-critical codes that must avoid certain runtime dependencies, or on systems where library compatibility is difficult to guarantee.

At compile and link time, you control static versus shared linkage through compiler flags and by choosing which libraries are present in your library paths. Some sites have specific policies about whether to link statically or dynamically for certain components, especially MPI or math libraries.

ABI Compatibility and Dependency Trees

In a large HPC software stack, libraries depend on other libraries, and applications depend on particular combinations of compiler versions, MPI implementations, and math libraries. This creates a dependency tree, where breaking one leaf or branch can affect many applications.

ABI, or Application Binary Interface, compatibility refers to the ability of compiled objects to interoperate at the binary level. If you compile a library with one compiler version and an application with a different, incompatible compiler, you might get subtle runtime failures. The same can happen if you mix different versions of MPI or different OpenMP runtimes in a single executable.

For this reason, HPC installations often standardize on a few compiler families and MPI implementations. Environment modules or container images encapsulate consistent combinations. When you install user software, you should align your choices with what your site supports. For example, if the cluster recommends a specific combination of GCC, OpenMPI, and a BLAS library, you should compile your software with the same combination.

Conceptually, you can think of each software stack as a tree rooted at the compiler and system libraries, branching into MPI, math libraries, I/O libraries, and finally applications. When you choose to install something yourself, you are creating a new subtree. Ensuring that this subtree is coherent and matches the environment you load at runtime is crucial for stable and reproducible behavior.

Installing Language-specific Packages

Purely interpreted or semi-interpreted languages such as Python, R, and Julia have their own notions of software installation. On HPC systems, language-specific package managers are heavily used, but with a few additional considerations.

For Python, tools like pip and environments such as virtualenv or conda install packages into environment specific directories. On an HPC cluster, you typically use pip install --user or create isolated environments in a project directory. Many scientific Python packages include compiled extensions that must be built against the correct compilers and libraries. This means that before installing them, you often need to load appropriate modules for compilers, MPI, or linear algebra libraries.

For R, you install packages into a personal library directory, often under $HOME/R/x.y/library. The command install.packages() uses CRAN or other repositories. On HPC systems, compiling R packages that depend on MPI or BLAS may require environment adjustments similar to Python.

For Julia and other languages with package ecosystems, the general idea is the same. The language runtime maintains a per environment package directory, and you must ensure that any compiled components are built against the cluster compilers and libraries, not just against generic system defaults.

In all these cases, the concept of a software prefix still applies, but it is managed by the language tools rather than by generic build systems. The cluster environment interacts with these tools through loaded modules, environment variables, and possibly site-specific configuration.

Install Locations and Filesystem Considerations

HPC clusters usually provide several categories of storage: home directories, project or scratch filesystems, and possibly local disks on compute nodes. Where you install software affects both performance and reliability.

Home directories are often backed by networked storage that may have strict quotas and backup policies. Installing small tools and scripts into your home directory is convenient and safe. However, compiling large software packages there can stress quotas and may be slower.

Project filesystems are designed for larger volumes of data and often for higher throughput. Installing large software stacks here can be more appropriate. They are also easier to share within a project team. However, they may have lifetime policies tied to the project, so you should consider long-term persistence.

Scratch filesystems are temporary and often purged automatically. They are optimized for high-performance I/O during large jobs. They are generally not suitable for permanent software installation, because you risk losing the installation without notice.

Local disks on compute nodes offer fast I/O but are typically not shared between nodes. For many job workflows, installing software on each node separately is impractical. Instead, administrators may stage certain components to local caches automatically for performance.

An important practical concept is that software installed on a parallel filesystem will be accessible on all nodes, which is usually what you want for compiled applications and libraries. However, initial load times can be higher when many nodes access the same files at once. System administrators may use techniques like preloading or caching to mitigate this, but as a user, you should at least know that the location of your installation affects how jobs start and perform.

Versioning and Coexistence

HPC environments must support multiple versions of the same software at once. Different projects may depend on different compiler versions or MPI stacks, and upgrading a library in place could break existing users. As a result, versioning and coexistence are central concepts in software installation.

At the system level, different versions are usually installed in separate directories, for example /opt/software/app/1.2.0 and /opt/software/app/2.0.1. Environment modules then adjust the PATH and related variables to point to the desired version. For user-level installations, you can adopt a similar convention, such as $HOME/software/app/1.2 and $HOME/software/app/1.3.

Instead of overwriting old versions, you typically install new versions side by side. This allows you to test the new version with existing workflows and roll back easily if needed. In some cases, symbolic links provide a default version, such as app/current pointing to the preferred release.

Understanding that multiple versions can and must coexist is important when you design your own per user software layout or when you request new software from HPC support.

Reproducibility and Recording Build Information

Installing software in HPC is not just a one time event. You must often reproduce builds later, perhaps on a new cluster, for a collaborator, or after a system upgrade. Reproducibility starts with recording what you did.

At a minimum, you should keep a record of: the source version and origin, for example a git commit or release tarball name; the compiler versions and options used; the modules that were loaded during the build; and the configuration options, such as --prefix or feature flags.

Some build systems can generate configuration reports automatically, or you can store your build scripts under version control. Language specific environment files, such as environment.yml for conda or requirements.txt for Python, serve a similar purpose.

In HPC environments, system administrators often document the build recipes for centrally installed software to ensure that they can rebuild the same versions after hardware or OS changes. As a user, mirroring this practice in a lighter form will save you time and reduce confusion when something needs to be rebuilt or moved.

Interacting with HPC Support for Installations

Finally, it is common on shared HPC systems to collaborate with the support team when you need new software. Many sites encourage users to request system-wide installations, especially for libraries or applications that are likely to be widely used.

From a conceptual standpoint, an effective request describes the purpose of the software, the required version, any specific compiler or MPI requirements, and the licensing conditions. It also helps to indicate whether GPU support is needed or whether the application needs particular math or I/O libraries.

Support teams then decide where in the software stack to place the installation, which compiler and MPI combination to use, and how to expose it through modules or other environment mechanisms. If your needs are experimental or highly customized, they may instead recommend that you perform a user-level installation.

Understanding the underlying concepts of prefixes, dependency trees, compilers, and filesystems will help you communicate effectively with support and interpret their choices when you use the installed software.

In summary, software installation in an HPC environment revolves around a consistent set of ideas: installation prefixes, environment configuration, build and link processes, dependency management, filesystem placement, versioning, and reproducibility. Once you recognize these patterns, you can navigate both system-wide and user-level installations with confidence, and you will be better prepared for the more advanced topics on software stacks, containers, and environment modules that appear later in this course.