Introduction to Singularity and Apptainer

What Are Singularity and Apptainer?

Singularity and Apptainer are container technologies designed specifically for HPC and scientific computing. They provide:

• Portable environments: bundle exact software versions, dependencies, and runtimes.
• User-level containers: typically no need for root access to run containers.
• Scheduler-friendly integration: work smoothly with SLURM, MPI, GPUs, and parallel filesystems.

Historically:

• Singularity was created for HPC use.
• The project was later donated to the Linux Foundation and continued as Apptainer.
• Modern HPC centers may:
• Use Apptainer (the upstream/open project),
• Use SingularityCE (community edition),
• Or both. For everyday use, commands and concepts are nearly identical.

In practice, you’ll often see:

• Executable command: singularity or apptainer
• Conceptual model: the same.

When using a real system, always check:

• The available module (module avail singularity or module avail apptainer).
• The local documentation for small differences (e.g., available subcommands, versions).

Why Containers in HPC (Specifically Singularity/Apptainer)?

Compared to general container tools (like Docker), Singularity/Apptainer are designed around HPC constraints:

• No root inside containers: by default, the container runs as you, not as root. This is important for security on shared systems.
• Access to host resources:
• Home directory (often mounted by default).
• Shared scratch/storage and parallel filesystems.
• Host network, GPUs, and MPI libraries (when configured appropriately).
• Integration with schedulers: you submit SLURM jobs that run containers just like other applications.

This makes it practical to:

• Develop your environment locally (or with Docker),
• Then run the same software environment on the HPC system using Singularity/Apptainer images.

Basic Concepts and Terminology

• Container image: a single file (often .sif – Singularity Image Format) which holds the full filesystem for the containerized environment.
• Definition file (.def): a recipe that describes how to build an image:
• Base OS image (e.g., Ubuntu, Rocky Linux).
• Packages, libraries, and custom build steps.
• Pulling: downloading an existing image from a registry (e.g., Docker Hub).
• Building: creating your own image from a definition file or from another container format (e.g., Docker).

Typical command structure:

• singularity pull ... or apptainer pull ...
• singularity build ... or apptainer build ...
• singularity exec ... or apptainer exec ...
• singularity run ... or apptainer run ...
• singularity shell ... or apptainer shell ...

In examples below, replace singularity with apptainer if that’s what your system provides.

Running Containers on HPC Systems

Running a container in HPC usually happens inside a batch job, but the core usage is the same on a login node (for quick testing).

Executing a Command in a Container

Use exec to run a single command inside a container image:

singularity exec myimage.sif python script.py

Key aspects:

• The current working directory on the host is also visible in the container (unless you override it).
• By default, your $HOME and some paths are bind-mounted; policies may vary by cluster.
• Your user ID is preserved: inside the container you are the same user as on the host.

You can also run interactive shells:

singularity shell myimage.sif

This drops you into a shell inside the container environment, useful for exploration and debugging.

Entry Points and `run`

If the container defines a “run script” (an entry point), you can use:

singularity run myimage.sif

This will execute the container’s default command (for example, launch an application script). The behavior is defined when the image is built.

Getting Images: Pulling from Registries

Singularity/Apptainer can directly use images from common registries (e.g. Docker Hub, Sylabs Cloud, GitLab registries).

Typical example (from Docker Hub):

singularity pull docker://python:3.11

This:

1. Downloads the Docker image.
2. Converts it into a Singularity Image Format (SIF) file, e.g. python_3.11.sif.

You can then run:

singularity exec python_3.11.sif python --version

Other URI schemes you may see:

• docker:// – Docker Hub or other container registries.
• library:// – Sylabs Cloud library (if your site uses it).
• oras:// – OCI registries (in newer Apptainer).

On many HPC systems, building or pulling large images is discouraged on login nodes; some centers provide prebuilt images or separate build nodes. Always check site policies.

Creating Custom Images (Overview)

Detailed build instructions typically require admin or root access (especially on personal machines), but the basic workflow is:

1. Write a definition file, e.g. myenv.def:

   Bootstrap: docker
   From: ubuntu:22.04
   %post
       apt-get update && apt-get install -y \
           python3 python3-pip
       pip3 install numpy
   %environment
       export OMP_NUM_THREADS=1
   %runscript
       python3 "$@"

2. Build the image (usually on a machine where you have appropriate permissions):

   sudo singularity build myenv.sif myenv.def

3. Transfer the .sif file to the HPC system and use it:

   singularity exec myenv.sif python3 -c "import numpy; print(numpy.__version__)"

On shared HPC systems, direct root-based builds are normally not allowed. Common patterns include:

• Building on a local workstation or in the cloud (with root), then copying .sif to the cluster.
• Using site-provided build services or dedicated build nodes.
• Using --fakeroot on systems that enable user namespaces (Apptainer feature; site-specific).

The details of building and distributing images are cluster-specific and are usually documented by the HPC center.

Bind Mounts and Accessing Data

For HPC work, data almost always lives on host filesystems (parallel filesystems, project shares, scratch). You typically do not bake large datasets into container images.

Singularity/Apptainer provides bind mounts so the container can see host paths:

singularity exec \
  --bind /project/mydata:/data \
  myenv.sif \
  python script.py --input /data/input.dat

Important points:

• --bind host_path:container_path
• The host path must exist and be accessible to you.
• Multiple --bind options can be specified.
• Your containerized application can read/write to these paths as usual.

On many clusters, some common paths ($HOME, project directories, scratch) are automatically bound; check your site’s configuration.

Using Containers with SLURM (High-Level View)

You can combine containers with SLURM job scripts. A minimal example:

#!/bin/bash
#SBATCH --job-name=container_test
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
module load singularity
singularity exec myenv.sif python script.py

Key ideas for HPC integration:

• Resource requests (CPUs, memory, GPUs) are handled by SLURM as usual.
• The container simply runs inside the allocated resources.
• For MPI jobs, you typically launch MPI from the host and call singularity exec inside the mpirun or srun command; exact patterns depend on the cluster’s MPI and container integration.
• For GPU jobs, you often need flags like --nv (for NVIDIA GPUs) to make host GPU drivers and devices visible:

  singularity exec --nv mygpuimage.sif ./my_gpu_program

Use the pattern recommended by your HPC documentation for MPI and GPU usage with Singularity/Apptainer.

Singularity vs. Apptainer: Practical Differences

From a beginner’s point of view, key differences are mainly names and versions:

• Command name:
• Some systems: singularity
• Others: apptainer
• Project governance:
• Apptainer is the current Linux Foundation–hosted project.
• SingularityCE is maintained separately; both remain similar in user interface.
• Configuration and features:
• Some advanced features (e.g., security options, --fakeroot, registry support) may behave slightly differently depending on version and build options.

In practice:

• Learn the generic syntax.
• Check singularity --version or apptainer --version.
• Adapt to small differences documented by your site.

Benefits for Reproducibility

In the broader context of reproducible workflows, Singularity/Apptainer help you:

• Pin software versions: record exact compiler, libraries, and tools in a single image.
• Capture environments as files: the .sif file is an artifact you can archive, share, and cite.
• Avoid “module drift”: your results no longer depend on which version of a module is currently installed on the cluster.
• Share with collaborators: everyone runs the same container image, even on different clusters.

Combining containers with good documentation and version control (for your code and definition files) makes it significantly easier to reproduce results months or years later, or on another HPC system.

Common Usage Patterns for Beginners

Typical early use cases include:

• Using a prebuilt image for a scientific package (e.g., GROMACS, TensorFlow) supplied by:
• Your HPC center, or
• A project’s official container.
• Creating a simple environment for Python/R where:
• You manage packages inside the container,
• You avoid conflicts with system-wide modules.
• Pinning a complete stack for a project:
• Create one image per project or per major workflow.
• Store images alongside the project in a shared location.

As you gain experience, you can gradually move from:

• Pulling existing images,
• To customizing images with your own definition files,
• To building robust, versioned environments that you can reuse across projects and systems.