KAHIBARO
41.5. Running Multiple Simulations
Table of Contents
Job arrays
On high performance computing systems you rarely submit many similar simulations one by one. Instead, you bundle them into a single job array. A job array is a group of jobs that share the same script and resources, but differ by an index. This is ideal for Monte Carlo studies where you need multiple independent runs, parameter sweeps, or repeated simulations for statistics.
Conceptually, a job array is defined by a job script that contains your usual environment setup and Python GATE command, plus one array index variable provided by the scheduler. For Slurm, this index is $SLURM_ARRAY_TASK_ID. Each array element runs the same script, but you can use the index to change input parameters, seeds, or output directories.
You typically use job arrays for three common patterns in GATE:
- Repeated runs of the same setup to improve statistics. You keep the same geometry, physics, and source, but split the total number of events or acquisition time across many independent runs. Later, you merge the results.
- Parameter scans. You vary one or more parameters across array indices, for example source position, energy, material thickness, or detector configuration. Each index corresponds to one parameter set.
- Patient or case batches. Each array element runs the same simulation logic on a different input dataset, such as different CT images or treatment plans.
Inside your job script, you normally convert the array index to something meaningful. For example, you can map task_id = int(os.environ["SLURM_ARRAY_TASK_ID"]) to a list of parameter values loaded from a configuration file, or use it directly to select a line from a text file with input definitions. For beginners, a simple approach is to use the array index as a run number that affects the random seed and output directory name.
A typical workflow on an HPC system looks like this. First, you ensure your GATE Python script reads configuration values from the command line or from a small JSON/YAML file that is specific to each run. Second, you write a Slurm job script that sets up the environment, activates your Python environment, and then calls Python with arguments that depend on the array index. Third, you submit the array with a command such as sbatch --array=0-9 job_array.sh so that ten simulations run independently. The scheduler decides when and where each array task runs, and you do not need to track them manually.
When using job arrays for large studies, it is important to separate output by array index. Use the index to create distinct output directories and filenames, for example output/run_${SLURM_ARRAY_TASK_ID}. This prevents different tasks from writing to the same file, which would corrupt your results. It also makes it easier to collect and merge data later, for example combining ROOT files or dose images.
Finally, job arrays help you respect HPC usage rules. Instead of asking for a single very large job with many CPUs, you can request many modest jobs that are easier for the scheduler to place. This usually results in better overall throughput for Monte Carlo campaigns that consist of many independent simulations.
Different random seeds
Monte Carlo simulations rely on random numbers. To obtain independent and statistically meaningful results when you run multiple simulations, you must control the random seeds. On HPC systems, job arrays simplify this because each array index can be mapped to a different seed. If you forget to change the seed between runs, you can end up effectively repeating the same simulation, even if file names or job IDs differ.
In GATE with Python, you typically set the random seed at the start of your script using a configuration parameter or a function call. Your job script then passes a unique seed to each run, for example through a command line argument derived from the array index. Inside the Python simulation script, you read that argument and configure GATE’s random engine accordingly, and, if needed, also set seeds for NumPy or other libraries that generate random numbers.
A simple pattern is to define the seed as a base value plus the array index. For example, you can choose a base seed like 100000 and then compute seed = 100000 + task_id where task_id is the array index. This guarantees that each job in the array uses a different seed while keeping the mapping simple and reproducible. You can record this seed in your output metadata so that any run can be reproduced later.
Always ensure that:
- Each simulation run uses a unique random seed, and
- The seed is recorded with the output for reproducibility.
Using different random seeds has two main purposes in GATE on HPC. First, it enables variance reduction by combining many independent runs. Each run samples different random histories and when you merge the results, the statistical uncertainty decreases roughly like $1/\sqrt{N}$ where $N$ is the number of independent runs. Second, it supports reproducible research. If you know the seed and the full configuration, you can re-run exactly the same simulation in the future.
When you design your simulation for HPC, plan explicitly how seeds are handled. Avoid letting each job pick a seed randomly at runtime without recording it, because that makes the simulation irreproducible. Instead, make seeding part of your controlled configuration. For example, maintain a table or small configuration file where each row corresponds to one array index and contains the seed and other parameters. The simulation can read this at startup, and your analysis can later refer to it.
Finally, when running very large numbers of simulations, be careful about seed ranges and integer limits, especially if you derive seeds from combinations of indices or timestamps. Keep the mapping simple, deterministic, and well documented so that you and others can understand and reproduce the exact set of runs that were executed on the HPC system.
Views: 11