Containers in HPC

Why Containers Matter for HPC Reproducibility

Containers help you package software, dependencies, and environment configuration into a portable unit that behaves consistently across different systems. In HPC, this addresses several common pain points:

• The cluster’s installed software is older or different from what you need.
• You develop on a laptop or workstation but run on a remote supercomputer.
• You want to share exact environments with collaborators or for publications.
• System administrators are (rightly) cautious about letting users install arbitrary software on login/compute nodes.

By isolating user-space (libraries, compilers, tools) while sharing the host kernel and hardware, containers let you reproduce complex software stacks without requiring admin privileges on the HPC system itself (with the right container runtime).

Container Basics in an HPC Context

A container in HPC typically provides:

• A well-defined Linux user-space (e.g., Ubuntu, CentOS, Rocky Linux).
• Specific compilers, libraries (e.g., MPI, BLAS, Python), and tools.
• Application code and scripts.
• Configuration for environment variables and entrypoints.

Key properties relevant to HPC:

• Isolation of user-space, not hardware: You still use the host CPU, memory, interconnect, and usually the host kernel’s drivers.
• Image immutability: Container images are usually read-only, making them ideal for reproducible, versioned environments.
• Separation of build and run: You can build images elsewhere (e.g., a cloud machine) and run them on a locked-down cluster.

Challenges of Containers on Traditional HPC Systems

Traditional container systems (like Docker) assume:

• You can run a daemon with elevated privileges.
• You have full control of the host machine.

On shared HPC clusters, that’s usually not allowed. Main challenges include:

• Security: Root in a container mapping to root on the host is dangerous in multi-user environments.
• Integration with schedulers: Containers must play nicely with job schedulers (e.g., SLURM), file systems, and accounting.
• Performance: MPI, high-speed interconnects, and parallel filesystems must work correctly inside containers with minimal overhead.
• GPU access: Containerized jobs still need direct access to accelerators (e.g., via NVIDIA drivers).

These challenges led to HPC-focused container runtimes with a different security and integration model.

HPC-Oriented Container Runtimes

Singularity / Apptainer (overview only)

In HPC, Singularity (now continued as Apptainer) became popular because it:

• Runs containers as the invoking user (no setuid root operations at runtime).
• Integrates well with shared filesystems and batch schedulers.
• Emphasizes compatibility with Docker images while being deployable safely on multi-user clusters.

The dedicated chapter on Singularity/Apptainer will cover usage; here the focus is why such tools exist and what makes them suitable for HPC.

Other Runtimes in HPC Environments

Depending on the site, you may encounter:

• Shifter: Designed by NERSC for large systems; similar goals of security and scalability.
• Charliecloud: Uses unprivileged user namespaces and focuses on simplicity.
• Podman / rootless Docker: Sometimes used on development clusters or single-user nodes.

The common theme: enable containerized workflows without requiring users to control system-level daemons or have administrator rights.

Containers vs. Environment Modules vs. Bare-Metal Installs

On HPC systems you’ll usually see three approaches to software environments:

1. Bare-metal + modules only
• Site provides compilers, MPI, and libraries as modules.
• You load what you need: module load gcc/12.2 openmpi/5.0.
• Good for: standard, well-supported software stacks.
• Limitation: you’re constrained by what the site installs and how it’s configured.
2. Containers only
• Your entire user-space is inside a container image.
• You may still use minimal host modules for MPI or drivers.
• Good for: distributing fully self-contained workflows, ensuring bitwise repeatability of the environment.
3. Hybrid approach
• Core system tools (MPI, drivers, vendor libraries) from the host.
• Application stack and Python/R ecosystem in a container.
• Good for: maximizing performance and compatibility while still having easy-to-reproduce user-level stacks.

For reproducibility, containers are particularly useful when:

• Dependencies are complex or fragile (scientific Python, machine learning, mixed-language codes).
• You expect long-term reuse of your environment (multi-year projects, archival of workflows).

Reproducibility Benefits Specific to HPC

Containers support several reproducibility guarantees that are especially meaningful in HPC:

• Captured dependency graph: The image encodes exact versions of compilers, libraries, and tools used at runtime.
• Decoupling from cluster upgrades: Even if the cluster OS or module tree changes, your containerized environment remains consistent.
• Portable performance experiments: You can run the same containerized code:
• On your laptop.
• On a development cluster.
• On a large production system.
  and compare performance with confidence that software differences are minimized.

Combined with job scripts and environment capture (covered under reproducible workflows), containers can make it much easier to:

• Re-run old simulations.
• Validate results reported in papers.
• Share workflows across institutions with different HPC stacks.

Containers and Performance in HPC

Performance concerns are central in HPC. For containers, key considerations include:

• Overhead: Properly configured, container runtime overhead is usually negligible for CPU- and memory-bound codes because:
• Containers share the same kernel.
• There is no hardware virtualization layer.
• MPI interoperability:
• Must ensure that containerized MPI stacks are compatible with host MPI runtime and network drivers.
• Sites may recommend:
• Using the host’s MPI libraries from inside the container, or
• Building containers that are ABI-compatible with the site’s MPI.
• I/O performance:
• Containers typically mount host filesystems. Parallel file systems (e.g., Lustre, GPFS) behave similarly inside and outside containers.
• Performance issues often arise from misconfigured mounts or non-optimal I/O patterns, not the container itself.
• GPU access:
• Requires exposing GPU devices and drivers into the container.
• Site-provided guidelines are essential (e.g., special flags or pre-built base images).

Performance tuning techniques (e.g., vectorization, profiling) apply equally in and out of containers; what changes is how you manage and reproduce the software stack.

Typical Container Workflows in HPC

Development vs. Production

A common pattern:

1. Develop and build image locally or in the cloud:
• Use Docker or rootless Podman to iterate quickly.
• Version the container configuration (e.g., Dockerfile, definition file) in git.
2. Convert or pull image to HPC environment:
• Use site-supported tools (e.g., Apptainer) that can:
• Pull Docker/OCI images.
• Build HPC-native image formats (often single .sif or similar files).
3. Run under the job scheduler:
• Wrap container invocation inside a batch script.
• Use scheduler options to request resources (nodes, tasks, GPUs) as usual.
• Inside the container, run the application with the expected MPI or GPU commands.

This separation allows you to do heavy software installation once, then re-use the resulting image across many runs and systems.

Sharing and Archiving Environments

For long-term reproducibility:

• Store container images in:
• Institutional container registries.
• Public registries (if appropriate for your project).
• Long-term storage (along with checksums and metadata).
• Document:
• Image name and tag.
• Exact command used to run the container in the job script.
• Relevant runtime options (e.g., bound directories, MPI configuration).
• Couple the image with:
• Version-controlled code and scripts.
• Configuration files and input data descriptions.

Together, these make it realistic for someone to reconstruct your environment years later.

Security and Policy Considerations

Because HPC systems are shared, containers must respect:

• User identity: Inside the container, you should run as your normal user, not as root.
• Filesystem permissions: Containers must not circumvent file access controls on shared storage.
• Network and scheduler integration: Containers must not spawn uncontrolled processes or evade resource limits.

As a result, many HPC centers:

• Prohibit user-run Docker daemons.
• Provide an approved runtime (e.g., Apptainer) integrated with their security model.
• Offer guidelines on:
• Where to store container images.
• Which base images to use.
• How to handle external network access from within containers (often restricted on compute nodes).

Always follow site-specific policies; “works on my laptop” approaches with rootful Docker often cannot be used directly on production clusters.

Best Practices for Using Containers in HPC for Reproducibility

When you start using containers to support reproducible HPC workflows:

• Prefer definition files over ad-hoc builds:
• Keep Dockerfile or Apptainer definition file in version control.
• Treat images as build artifacts, not hand-edited environments.
• Pin versions where it matters:
• Use explicit versions for base images, compilers, MPI, Python packages, etc.
• Avoid floating tags like latest for research workflows you need to reproduce.
• Keep images lean but complete:
• Include all tools needed to build and run your application.
• Remove unnecessary packages that complicate dependency graphs and enlarge images.
• Match site recommendations:
• Use base images or MPI stacks suggested by your HPC center to avoid incompatibilities.
• Follow their recipes for GPU access and parallel filesystem mounts.
• Record runtime context:
• Along with the image tag, record:
• Scheduler options (nodes, tasks, memory).
• Environment variables or modules (if any) needed on the host side.
• Container runtime flags (bind mounts, MPI integration options).

By combining containers with good documentation and workflow management, you make your HPC work far easier to reproduce, validate, and share.