41.4. Running with Slurm

Table of Contents

CPU allocation

On most high performance computing systems, Slurm manages how many CPUs your GATE job can use. For GATE, this usually means controlling the number of threads in a multithreaded simulation, or the number of independent single-threaded jobs.

You request CPUs in your Slurm script. A typical script for a single multithreaded GATE job looks like:

bash

#!/bin/bash
#SBATCH --job-name=gate_pet
#SBATCH --cpus-per-task=8
#SBATCH --ntasks=1
module load python              # or your site-specific modules
source ~/envs/gate/bin/activate # your Python environment
python run_pet_simulation.py --n_threads 8

Here --cpus-per-task=8 asks Slurm for 8 CPU cores for a single task. Inside your Python script you must configure GATE to use the same number of threads. For OpenGATE this is usually something like:

python

sim = gate.Simulation()
sim.number_of_threads = 8

Always match the GATE thread count to the CPUs you request from Slurm. If you use more threads than allocated CPUs, your job can slow down the whole node and may be killed by the system administrator. If you use fewer threads than CPUs, you waste resources and your simulation is slower than it could be.

On some clusters, administrators prefer that you request resources with --ntasks and --cpus-per-task together. For a single multithreaded job, the standard pattern is:

bash

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16

For running many independent simulations at once, you have two main patterns.

First, many tasks on one node:

bash

#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1

Then inside your script each task runs a separate simulation with one thread. This is often used with job arrays, explained in a separate chapter.

Second, one task per Slurm job in a job array. In that case each script uses:

bash

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4

and you choose sim.number_of_threads = 4 in Python. Each array element runs independently with its own random seed and parameter set.

Always check with your local documentation whether your site wants you to use --cpus-per-task or --ntasks-per-node. The basic idea remains the same: the total number of cores you request must equal the number of GATE threads you intend to run on that node.

Memory allocation

Slurm can also control how much memory your GATE jobs may use. If you do not specify memory, the system assigns a default per core. For complex GATE geometries or large dose grids, this default can be too small, so it is safer to request memory explicitly.

The two common options are memory per node and memory per CPU:

bash

#SBATCH --mem=32G          # 32 GB for the whole job

bash

#SBATCH --mem-per-cpu=4G   # 4 GB per allocated CPU

If you request --cpus-per-task=8 and --mem-per-cpu=4G, Slurm will allocate 32 GB of memory to your job.

For most small to medium GATE simulations, 2 to 4 GB per CPU is enough. For voxelized patient geometries or many dose actors, you may need 8 GB per CPU or more. You can estimate memory needs by running the simulation on a local machine or with an interactive short job and using system tools to watch memory usage while the simulation is running.

If you request too little memory, your job may be killed by the operating system when it reaches the memory limit. This often appears as your job ending early without a clear Python error. If you request far too much memory, Slurm may not start your job for a long time because there is no node with enough free memory, and you block resources that others could use.

A simple rule is to start with a conservative memory request, check whether the job runs to completion without being killed, then adjust. Many clusters provide job statistics after completion, for example with a command like:

bash

seff <job_id>

that shows the maximum memory used by your job. You can then reduce or increase your next memory request to match this value with a reasonable safety margin.

When you run job arrays with many simultaneous jobs, sum the memory per job to know how much memory one node must provide. If the node has 128 GB and you start 16 jobs each requesting 8 GB, you will probably overload the node and your jobs will not start. Adjust the number of simultaneous jobs or the requested memory so that the total fits within node limits.

Job duration

Every Slurm job must specify a maximum run time. You set this with the --time option:

bash

#SBATCH --time=02:00:00    # 2 hours

The format is days-hours:minutes:seconds or hours:minutes:seconds. If your job reaches this limit, Slurm will stop it even if the simulation is not finished.

To choose a good time limit you need a rough estimate of how long your GATE simulation will take. For beginners, a practical method is:

Run a short test simulation with fewer events or shorter acquisition time and measure how long it takes. For example run 1e5 events and note that it takes 5 minutes.
Scale this to your full job. If you plan to run 2e6 events, the time is roughly 20 times larger, so about 100 minutes.
Add a safety factor, for example 1.5, and request about 150 minutes, which is 2 hours and 30 minutes.

You then set:

bash

#SBATCH --time=02:30:00

If your job exceeds the requested time, Slurm will cancel it and you may lose partial results, especially if you do not write intermediate output. If you request a very long time unnecessarily, your job can wait much longer in the queue because the scheduler finds it harder to fit a long job into the cluster schedule.

Many centers have partitions with different maximum times, for example one partition for short jobs up to 2 hours and another for long jobs up to 48 hours. For GATE, it is often better to split a very long simulation into several shorter runs, each with its own random seed, and combine the results afterwards. This fits better into typical HPC queues and reduces the risk of losing all progress if one job is cancelled.

Make sure that the simulated acquisition time inside GATE is independent of the Slurm wall time. For example, you can simulate 1 hour of PET acquisition using 10 minutes of CPU time if the number of events is small. The GATE acquisition time is a physics parameter, while the Slurm --time limit is a resource parameter. They are not the same quantity.

After your job finishes, use Slurm accounting tools to see how much of the allocated time you actually used. If your job consistently finishes much earlier than the time limit, you can reduce --time to improve queue position. If your job often reaches close to the limit, consider increasing it a bit or reducing the number of simulated events.

41.4. Running with Slurm

CPU allocation

Memory allocation

Job duration

Comments

Where to Move