Table of Contents
Understanding Filesystems on HPC Systems
On an HPC system you interact with data through a filesystem. Every file you read or write, every program you run, and every script you submit to the scheduler lives in some part of a tree of directories. For efficient and safe work on a cluster it is not enough to know that “there are files.” You must know where they live, who owns them, which parts are shared, and which are fast or slow. This chapter focuses on the practical structure of filesystems and directories as you meet them in a Linux-based HPC environment.
On shared HPC systems, where you put your files is as important as what they contain. Using the wrong filesystem can make your jobs extremely slow or can cause them to fail when quotas are exceeded.
The Linux Filesystem as a Single Tree
Linux presents all storage as a single tree that begins at the root directory /. There are no drive letters. Instead, every disk or network filesystem is attached, or “mounted,” somewhere under /.
From your point of view as a user, the most important consequence is that every object has one unique path, for example /home/alice/project/run1/input.dat. You navigate this tree using paths, and your current working directory acts as your reference point.
Absolute paths start at /, such as /home/alice. Relative paths start from your current directory, such as ../scripts to move to the parent’s scripts subdirectory.
HPC systems typically mount several distinct storage areas into this tree. Even though they look like ordinary directories, they may be physically separate storage systems with very different performance and policies. Understanding which is which is a core skill in cluster work.
Typical Directory Layout on HPC Clusters
While every site has its own conventions, many clusters follow similar patterns that you can recognize.
System-wide directories near the root hold operating system files and shared software. System administrators manage directories like /bin, /usr, /lib, and /opt. As a regular user you usually only read from these, for example when loading compilers or libraries needed for your applications.
User-related data is kept under paths such as /home, /users, or /cluster/home. Inside these, each user has a personal directory, sometimes called a home directory. You normally start in your home directory at login, and it is the default place for your personal scripts, configuration files, and small datasets.
HPC systems often add additional top-level directories for project storage and temporary or scratch storage. You might see names such as /project, /group, /work, or /scratch. Each of these is a separate filesystem with its own performance characteristics and policies. Your cluster documentation will state which ones are backed up, which have quotas, and which are meant only for high-speed temporary data.
Even though all of these appear as subdirectories of the root tree, it is helpful to think of them as different “kinds of space” tailored to specific usage patterns such as long-term storage, collaborative work, or fast scratch space for running jobs.
Home Directories and Their Role
Your home directory is the personal center of your filesystem world. In Linux this is usually /home/username, and you can always refer to it as ~ or with the environment variable $HOME. When you log in to an HPC cluster, the shell typically starts in this directory.
Home directories on clusters are typically:
- Relatively small, with storage quotas.
- Backed up regularly, so accidental deletion can sometimes be recovered.
- Mounted on all login and compute nodes so you can access your configuration files and scripts everywhere.
Because of quotas and performance, home directories are not usually the right place for large datasets or temporary output from large simulations. They are, however, the right place for source code, submission scripts, build directories, and configuration files like .bashrc or .profile that customize your environment.
Permissions on your home directory are normally set so that only you and system administrators can read its contents. Within your home directory you may create subdirectories such as src, data, or jobs to keep your work organized, but how you structure it is up to you and any project conventions you follow.
Project, Group, and Shared Directories
In collaborative HPC workflows you rarely work completely alone. Many clusters therefore provide space dedicated to research groups or projects. Such directories may be named /project, /projects, /group, /share, or similar, with subdirectories like /project/mygroup or /project/mygroup/user1.
These shared areas usually have the following properties:
- They are accessible by multiple users from the same project or group.
- They are intended to store data and software that a team uses collectively.
- They may have a larger quota than home directories, or a quota applied per group.
Within these spaces, file permissions and group ownership determine who can see what. Often, your account belongs to one or more Unix groups corresponding to research teams. System administrators can then arrange that files in /project/mygroup are owned by that group so your collaborators can read and write them.
If you develop software or manage datasets that others will use, shared project directories are usually the right place. It is common, for example, to keep large reference datasets in a shared area rather than duplicating them in every user’s home.
Scratch and Temporary Filesystems
HPC workloads often produce or consume very large amounts of data. To support this, clusters provide scratch storage. Scratch filesystems might be mounted as /scratch, /work, /tmp, or node-local directories such as /local_scratch.
Scratch space is typically:
- High performance, sometimes parallel and tuned for throughput.
- Not backed up, so data may be lost if the system fails or files are purged.
- Subject to automatic deletion policies based on age or inactivity.
Because of these properties, scratch is ideal for files that are expensive to keep elsewhere but cheap to recompute or regenerate. Examples include intermediate results, checkpoint files between job steps, or expanded working copies of compressed input data.
Multiple scratch tiers can exist. For instance, a global scratch filesystem is visible from all nodes and is suitable for jobs that span many nodes, while node-local scratch exists only on a particular compute node and provides very fast access for that node alone. Node-local scratch disappears when the node is rebooted or your job ends, so it is a temporary workspace.
Scratch space is not for long-term storage. Do not rely on scratch for anything you cannot recreate. Copy your important results back to a backed-up filesystem, such as your home or project directory, before your job finishes or shortly after.
Quotas, Limits, and Policies
On shared clusters storage is a limited resource. Filesystem quotas and policies control how much space each user or group can consume and how long data can be kept in certain areas.
Quotas are typically applied in terms of used bytes and number of inodes, where each file and directory consumes one inode. You might have enough space in terms of gigabytes but run out of inodes if you create a very large number of tiny files. You will often have different quotas for home, project, and scratch filesystems.
In addition to quotas, scratch filesystems may implement purge policies. For example, files that have not been accessed for a certain number of days may be deleted automatically to free space. These policies are specific to the site and are usually documented and advertised.
You can usually check your usage with commands provided by your site or with general tools that report filesystem usage. Staying within quotas and respecting purge policies is part of responsible HPC use and helps prevent job failures due to “no space left on device” errors.
Directory Structure Conventions for HPC Work
Although Linux itself does not impose a specific layout for your personal and project directories, having a consistent pattern helps you and your collaborators manage complex work. In HPC, well-organized directories directly support reproducibility, automation, and debugging.
A common approach is to separate input data, code, job scripts, and output results into distinct subdirectories. For example, under a project directory you might have a structure where src holds code, inputs holds parameter files and small reference data, and results holds the outputs produced by jobs. Within results, further subdirectories may separate different experiments or parameter sweeps.
Another helpful convention is to include job identifiers, parameter values, or date stamps in directory names. This makes it easier to match directories to particular job submissions and to understand at a glance what each directory contains.
On the scratch filesystem it is typical to mirror enough of your project structure to make it clear which scratch subdirectory belongs to which project and run. For example, you might create a scratch directory that parallels part of your project path and holds only temporary working files for a particular campaign of runs.
System Directories You Encounter but Rarely Modify
Beyond home, project, and scratch spaces, the Linux filesystem tree contains many directories that support the operating system and installed software. While you do not usually write to these, it is helpful to recognize them when you see them.
Directories such as /bin, /sbin, /usr/bin, and /usr/sbin contain executables for commands you run. Libraries used by programs are placed under /lib, /lib64, or /usr/lib. System configuration files live in /etc. Device representations use /dev, and runtime information such as process identifiers and system statistics are under /proc and /sys.
On HPC clusters, additional software such as compilers, numerical libraries, and parallel tools may be installed under /opt or under site-specific trees such as /apps or /cm/shared. Environment modules, discussed elsewhere, help you select and use software from these directories without manually adjusting your PATH.
You should not attempt to modify system directories on shared clusters, since they are centrally maintained and often read-only for normal users.
Paths, Environment Variables, and Shortcuts
Efficient navigation in large directory trees depends on understanding paths and a few key environment variables. A path like /home/user/projectA/results/run01 tells you precisely where you are, but typing such long paths repeatedly is inconvenient. Linux provides shortcuts such as ~ for your home directory and .. for the parent directory. Shell features such as tab completion also help reduce typing and prevent errors.
Environment variables are placeholders that hold important paths. $HOME stores your home directory, $PWD your present working directory, and $TMPDIR may point to a job-specific temporary directory on scratch or node-local storage, especially when you run batch jobs. Many batch systems and cluster configurations set variables such as $SCRATCH or $WORK to direct you to site-recommended locations for large temporary files.
When preparing job scripts you usually specify filesystem paths for input and output. Using environment variables instead of hard-coded paths makes scripts more portable across clusters and user accounts. For example, output directories can be placed under $SCRATCH for performance while code lives under $HOME or a project directory.
In job scripts, prefer environment variables like $HOME, $SCRATCH, or $TMPDIR instead of absolute paths typed by hand. This reduces mistakes when you move between systems or accounts and helps your jobs follow site policies automatically.
Filesystem Layout and Job Performance
The choice of filesystem and directory for your data directly impacts how your jobs perform. Reading and writing large files from slow or heavily loaded filesystems can transform a short compute job into a long and inefficient one. Clusters usually provide documentation describing which filesystems to use for which workloads.
As a rule of thumb, small configuration files and scripts can safely live in your home or project area, but large input data and voluminous output should use high-performance filesystems, often under scratch. Parallel filesystems described elsewhere are designed to support simultaneous access from many processes and nodes and thus are the right target for distributed parallel jobs.
Node-local scratch may outperform global storage for temporary intermediate data that never needs to be shared between nodes. In that case, each rank or process can write its own intermediate files to local scratch and only move final results back to a shared filesystem at the end.
Attention to directory placement also helps avoid contention. If many users concurrently write to a single shared location, metadata operations can become a bottleneck. Spreading data across separate directories, especially on parallel filesystems, can mitigate these effects and improve job throughput.
Summary
In the Linux-based HPC environment, all storage is presented as a single directory tree, but below that unified view lies a variety of filesystems with different capacities, speeds, and policies. Your home directory is a personal, usually backed-up space best suited to code, scripts, and modest data. Shared project areas support collaboration, while scratch filesystems offer high throughput at the cost of persistence and backup.
Recognizing which directories correspond to which underlying storage, respecting quotas and purge policies, and organizing your own directories systematically are core practical skills. Good choices in filesystem and directory structure lead to more reliable, reproducible, and performant HPC workloads.