Table of Contents
Introduction
High performance computing systems are shared resources. Many users and many applications compete for the same CPUs, memory, and storage. Job scheduling and resource management are the mechanisms that make this sharing possible in a controlled and predictable way. Instead of users logging directly on to compute nodes and starting programs interactively, they describe what they need, when they need it, and how long they expect to use it. The scheduler then decides when and where to run each job.
In this chapter you will learn how modern HPC systems organize work, what a job really is in this context, how resources are represented and allocated, and how policies influence the performance you observe as a user. Later chapters will introduce concrete tools such as SLURM and job scripts. Here the focus is on the concepts that are common across most batch scheduling systems.
What a “Job” Means in HPC
On an HPC cluster, a job is a request to run some program with specified resources for a specified amount of time. Conceptually, you are making a contract with the system. You state that you want, for example, 64 CPU cores, 256 GB of memory, and access to a certain amount of storage, for up to 2 hours, to run a particular executable with given input data. In return, the scheduler promises to start your job at some point that satisfies both your request and the system’s policies.
A job has a life cycle. It is created when you submit it to the scheduler. While it waits for resources it is in a queue, often called the pending state. Once the scheduler assigns resources, the job enters the running state. After completion or failure it reaches a finished state, which may be success, error, or cancellation. The system typically records metadata about each job, such as its run time, resources used, and exit status, which are later used for accounting and performance analysis.
Jobs can be interactive or batch. Interactive jobs provide a shell or user interface on allocated resources, which you then use manually. Batch jobs run a predefined set of commands without further human interaction. Most production workloads in HPC are batch jobs, because they are easier to schedule and to automate.
A job is not just a process. It is a resource request plus a set of processes bound by scheduler policies and accounting.
Representation of Resources
To schedule jobs, the system must represent resources in a structured way. The most basic resource is the CPU core. Jobs often request a certain number of cores, either on a single node or across multiple nodes. In a shared memory context, a job may request a certain number of hardware threads instead of or in addition to cores, depending on the architecture.
Memory is another primary resource. The scheduler can track memory at the node level or per core. A job might request total memory, such as 64 GB on a node, or memory per task, such as 4 GB per process or thread. If the job exceeds its memory allocation, the system may terminate it to protect other jobs.
Nodes are also resources. A job may request entire nodes, for example 4 nodes with 32 cores each. This is typical for distributed memory applications. In such cases the scheduler ensures exclusive use of those nodes, or manages sharing according to configured policies.
Modern clusters have additional resources that are treated specially. GPUs and other accelerators are often represented as generic consumable resources. A job may ask for 2 GPUs on a particular node type. The scheduler must understand the mapping between GPUs, CPU cores, and NUMA domains in order to allocate them consistently.
Storage and I/O bandwidth can also be modeled as resources. Some schedulers support limits on file system capacity or I/O rates per job. Even network bandwidth can be considered, although this is less common in simple configurations.
Finally, there are logical properties such as node features or partitions. Features can include CPU generation, amount of memory, GPU model, or special interconnect topology. Partitions group nodes into sets with similar characteristics or policies. When you request a job in a particular partition or with certain features, the scheduler narrows the set of possible nodes to those that match.
Queues, Partitions, and Job Classes
Job schedulers organize work into queues or partitions. Though naming differs between systems, the idea is similar. A queue represents a subset of the cluster with a specific policy. Different queues may exist for short, medium, and long jobs. Some queues may be reserved for specific projects or for interactive use. Others may be used for high priority workloads such as urgent production jobs.
Each queue or partition has limits. These can include a maximum wall time per job, a maximum number of nodes per job, a maximum number of running jobs per user, or total core-hours available to a project. These constraints help prevent a single job or user from dominating the system and also help shape workloads into patterns that match the hardware.
Job classes or quality of service settings refine this idea further. They can encode priorities, preemption rights, or access to special resources. A high importance job class may be allowed to preempt lower importance jobs. A low importance class might be restricted to idle hardware and scheduled only when the system would otherwise be unused.
As a user, selecting the right queue or class is part of effective resource management. A job that can complete in 30 minutes often runs much earlier in a short queue than in a general purpose queue, because schedulers use these small jobs to fill gaps between larger allocations.
Always match your job’s wall time and resource needs to the queue or partition rules. Overestimating can reduce your priority and underestimating can lead to termination when the time limit is reached.
Wall Time and Job Duration
Time is a central concept in resource management. Schedulers need an estimate of how long your job will run, called the wall time limit. You specify a maximum wall clock duration, such as 1 hour or 48 hours. The scheduler uses this estimate for two reasons. It can enforce limits so that runaway jobs do not occupy resources indefinitely, and it can make backfilling decisions that depend on knowing how long jobs will run.
Backfilling is a technique where the scheduler temporarily fills idle slots with shorter jobs without delaying earlier queued jobs. These decisions rely on the requested wall time. If you request much more time than your job actually needs, the scheduler may hesitate to start it when there are gaps in the schedule, because it expects the job to run long and potentially affect the start time of a higher priority job. Accurate time estimates therefore help both you and other users.
Time is also the foundation of accounting. Systems monitor the product of resources allocated and time used, which gives a number such as core-hours or GPU-hours. These metrics are used to enforce project quotas and to generate usage reports. Your requested wall time, once combined with your resource selection, determines the potential cost of your job in terms of these accounting units.
Scheduling Policies and Priorities
Schedulers apply policies to decide which job runs first when resources become available. These policies try to balance fairness, system utilization, and the needs of important workloads. While implementations differ, common elements appear in many installations.
A primary factor is job priority. Each job receives a numeric priority that may depend on several components. One component is age, which represents how long the job has been waiting. Another is fair share, which measures how much of the system a user or project has consumed recently compared to what they are entitled to. A user who has used fewer resources than their allocation generally gains higher priority for their upcoming jobs.
There may be a component based on job size. Some systems favor larger jobs to encourage efficient use of whole nodes and minimize fragmentation. Others emphasize throughput of smaller jobs. Specific clusters may expose these details or hide them, leaving users only with practical guidelines.
Certain jobs run with elevated importance. This can be achieved with dedicated queues, special job classes, or explicit priority boosts. These jobs may also be allowed to preempt existing jobs. Preemption means the scheduler can suspend or terminate lower priority jobs to free resources for high priority work. In environments that support preemption, users are often steered toward checkpointing and restart mechanisms so that preempted jobs do not lose all progress.
From the user perspective, these internal details appear as patterns in waiting times. Large, long jobs often wait longer but then run uninterrupted once started. Short jobs may be scheduled quickly using backfilling. Job priorities may change over time as your project approaches or exceeds its fair share target.
Backfilling and System Utilization
Efficient use of an HPC system is only possible if resources are kept busy. Backfilling is one of the main strategies schedulers use to achieve high utilization without violating fairness. The scheduler maintains a set of reservations for top priority jobs, particularly large jobs that have been waiting. If some resources are temporarily free but are scheduled to be needed soon, the scheduler tries to fill that time window with shorter or smaller jobs.
To decide whether backfilling is safe, the scheduler uses your wall time estimate. It only starts backfill jobs that are expected to finish before reserved resources are needed. In practice, this means that jobs with modest resource requests and accurate time limits often experience very short queue times.
Backfilling affects the way you should think about your job submissions. Many small jobs are easier for the scheduler to weave into gaps in the schedule than a single very large request. However, the optimal strategy depends on the system’s configuration, since some schedulers reward larger, more scalable jobs. Reading local documentation and observing behavior over time can help you choose how to structure your workloads.
Underestimated wall time can cause your job to be killed at its time limit. Overestimated wall time can prevent your job from being used for backfilling and may increase its wait time.
Fair Share and Accounting
Clusters are often shared among multiple projects or departments. Fair share mechanisms try to ensure that each group receives access that matches its allocation over a longer period, such as weeks or months. The scheduler keeps track of historical usage and compares it to configured targets.
If your group has used less than its share, your jobs may receive a priority boost. If your group has consumed more than its share, your new jobs may be de-prioritized until other groups catch up. Fair share can be applied hierarchically, with policies at institutional, project, and individual user levels.
Accounting systems track each job’s consumption of resources. The primary measure is usually the product of allocated cores and elapsed wall time, called core-hours. For GPU workloads, GPU-hours may be tracked separately. Memory and storage usage may also be recorded, particularly if they are scarce or billed resources.
Some centers allocate a fixed number of core-hours or similar units to each project per allocation period. When the project runs out of allocation, its priority may be reduced, or it may be prevented from submitting further jobs until more allocation is granted. In other environments, usage is monitored for reporting and capacity planning rather than strict enforcement.
Understanding fair share and accounting helps explain cases where your jobs wait longer than expected. It also guides you to use resources efficiently, because underutilized allocations can be considered in future allocation decisions.
Resource Limits and Enforcement
To protect the stability of the system and maintain fairness, schedulers and resource managers enforce various limits. Some are static limits defined in the configuration. Others can be set by users when defining their jobs, subject to maximums.
Typical limits include the maximum number of running jobs per user, the maximum number of jobs in the queue, and the maximum total resources per user or project. These prevent excessive job submissions from overloading the scheduler or crowding out other users.
Within each job, the resource manager configures limits such as maximum address space, maximum CPU time per process, and maximum number of open files. These are often implemented using mechanisms such as ulimit at job start. If your code violates these boundaries, the operating system or resource manager can terminate it.
Time limits are particularly important. When a job reaches its wall time, the scheduler sends signals to its processes and ultimately kills them if they do not exit. Some schedulers provide warnings before termination, which you can use to initiate checkpointing. Memory limits also trigger enforcement. If a job exceeds its allocated memory, it can be stopped to avoid interference with other jobs sharing the node.
From a user’s standpoint, resource limits are constraints to respect when designing simulations and workflows. Choosing appropriate problem sizes, implementing checkpointing, and monitoring usage during test runs all help you operate within those boundaries.
Reservations, Maintenance, and Special Use
Schedulers can also create reservations that guarantee resources for specific uses. Reservations may be used for tutorials, workshops, or critical project deadlines. They block certain nodes for a particular time window so that jobs associated with the reservation can start immediately.
Reservations interact with ordinary scheduling policies. Jobs that are not part of the reservation cannot use reserved nodes during the reserved time, although they may run on them before the window begins if there is time to finish. This requires careful planning by system administrators to avoid low utilization.
Maintenance periods are another form of reservation. The scheduler blocks nodes or entire clusters during scheduled maintenance, which may include hardware repairs, software upgrades, or cooling system work. Jobs that would extend into maintenance windows are either not started or receive special treatment depending on local policy.
Some environments provide debug or testing partitions where time and size limits are very small, but queue times are short. These are valuable for rapid development and validation of job scripts without waiting in the main production queues.
User Responsibilities and Best Practices
Effective job scheduling is not only the responsibility of the system. Users play a crucial role by providing accurate information and by structuring their work in ways that align with policies. Mis-specified jobs can lead to poor system utilization, long wait times, and increased risk of failures.
A key responsibility is resource estimation. Before launching large production jobs, run small test cases to measure memory usage, run time, and scaling behavior. Use these measurements to infer appropriate core counts and wall time limits for larger jobs. Avoid the extremes of requesting far more resources than you can use efficiently or requesting too few and causing excessive run times.
Another responsibility is managing job arrays and workflows. Job arrays, which will be described in detail in later chapters, let you submit many similar jobs under a single umbrella. This is more efficient for both you and the scheduler than submitting thousands of separate jobs. Workflow tools can chain jobs and handle dependencies so that new jobs start when previous ones finish. This reduces idle gaps and manual intervention.
You should also monitor running jobs to ensure they are making progress and using resources as expected. If you detect a problem, cancel the job promptly rather than letting it occupy resources uselessly. Many centers log user behavior and may contact you if your jobs consistently fail or waste resources.
Use test runs to calibrate your resource requests. Cancel unproductive jobs quickly. Align your workflows with the scheduler’s features such as arrays and dependencies to help both your performance and the cluster’s efficiency.
Interaction Between Scheduling and Parallelism
Job scheduling and resource management are closely connected with parallel programming models like MPI and OpenMP. The resources you request at the job level define the limits for the parallel entities inside your application. If you request 4 nodes with 16 cores each, for instance, your MPI job must create no more than 64 processes to avoid oversubscription or interference.
Schedulers can bind processes to specific cores and nodes according to your job’s specification. They can also set environment variables that inform your application about the allocated resources, such as the number of tasks or rank identifiers. Parallel libraries rely on this information to build communication patterns and thread pools.
Hybrid applications that combine multiple forms of parallelism must be particularly careful. The mapping between MPI ranks, threads, and hardware resources should be consistent with the scheduler’s view. Incorrect mappings can reduce performance drastically, even if the total counts match.
As you develop parallel applications, you should always design with the scheduler in mind. This means considering how your code behaves when run across many nodes, how it recovers from failures or preemptions, and how it adapts to different resource shapes. Later chapters on MPI, OpenMP, and hybrid programming will build on this foundation.
Summary
Job scheduling and resource management provide the structure that makes shared use of HPC clusters possible. By treating work as jobs that request specific resources for a limited time, schedulers can enforce fairness, maximize utilization, and respect administrative policies. Concepts such as queues, priorities, backfilling, fair share, and reservations all serve to organize workload on complex systems.
For you as a user, understanding these ideas is essential to getting good performance and reliable access to resources. Accurate resource requests, sensible wall time limits, and thoughtful use of scheduler features will help your jobs start sooner, run more predictably, and make better use of the cluster. In the following chapters, you will learn about specific batch systems, including SLURM, and how to translate these concepts into concrete job scripts and commands.