KAHIBARO
40.6. Large Simulations
Table of Contents
Batch processing
When you move from small test cases to realistic medical physics problems in GATE, the number of events, the size of the geometry, and the complexity of the physics can easily push single simulations to hours or days of CPU time. At that point, you rarely run one huge job. Instead, you split the work into many smaller jobs that can run in parallel on a workstation or HPC system. This systematic splitting and automated execution is what we call batch processing.
In a batch workflow you do not start simulations interactively, and you do not rely on visualization. You prepare scripts and configuration files that completely define the simulation, then submit many jobs that execute without user interaction and write results to well-defined output directories.
A key ingredient of batch processing is a clear separation between your “simulation definition” and your “run configuration.” The simulation definition describes geometry, materials, physics, sources, actors, and digitizers. It should be as independent as possible from details like the number of events, random seed, or the specific output path. Those run-time details are then controlled by external parameters, for example environment variables, command line arguments, or small configuration files.
You can implement this in Python by reading parameters at the top of your script. For instance, you might read the number of events and a seed from the command line, then pass them into your simulation object and into your random number configuration. This way, the same script can be reused thousands of times with different configurations during batch processing.
A typical batch strategy for large simulations is to partition your total number of events into many jobs. Each job runs the same geometry and physics, but with a smaller event count and a different random seed. For example, if you need $10^9$ events, you might run 100 jobs with $10^7$ events each. This improves resilience because if one job crashes, you only lose a fraction of the statistics, and it allows the use of many cores or nodes in parallel.
Another important aspect of batch processing is output organization. For large simulations you will produce a significant number of files, so you should adopt a consistent directory structure and naming scheme. An effective pattern is to create one directory per job that contains all outputs for that job, along with a small metadata file describing configuration parameters such as the seed, number of events, and software version. You can encode the job index or seed into the folder name so that you can later reconstruct how each file was produced.
Use of logging also becomes crucial in batch processing. Since you do not see the terminal during execution, you should redirect standard output and error logs to files. You can configure GATE or your Python script to write a log per job, containing key information such as simulation start and end time, number of events processed, and any warnings. These logs are essential for debugging problems that arise only at large scale.
When you run many jobs concurrently, I/O and file size can become a bottleneck. To keep batch processing efficient, you should limit the amount of data per event. Only record what you truly need for your analysis. For instance, instead of writing full hits, singles, and coincidences for every event in a huge PET study, you might only record singles and a few summary actors, or use phase space data efficiently.
In large batch simulations, always:
- Run a small test with a tiny number of events and the exact same script and configuration.
- Verify output content and size before scaling to many jobs.
- Control random seeds explicitly so jobs are independent and reproducible.
Finally, batch processing is closely related to using HPC schedulers like Slurm, which are covered elsewhere. Regardless of the platform, the core ideas remain the same: noninteractive runs, parameterized scripts, controlled random seeds, and disciplined output management.
Multiple simulation runs
Large projects rarely rely on a single simulation result. Instead, you typically perform multiple simulation runs that explore different conditions, verify convergence, or provide uncertainty estimates.
One common reason for multiple runs is to improve statistical precision by combining independent simulations. Since Monte Carlo uncertainty scales roughly as $1/\sqrt{N}$, you can achieve the same total number of events by summing many smaller runs. In practical terms, you can run $K$ independent simulations, each with $N$ events, and then merge the outputs as if you had run one simulation with $K \times N$ events, provided that the random seeds and histories are independent and the physics and geometry are identical.
Another purpose of multiple runs is systematic parameter studies. You may want to scan over beam energies, source activities, detector configurations, or geometry dimensions. For each set of parameters, you run a separate simulation and compare the output. In such studies, you typically define a parameter grid, then generate one configuration for each point. With Python, you can automate this by looping over parameter values and writing distinct configuration files or passing parameters directly on the command line when launching each run.
To keep track of many runs, you need a clear naming and indexing strategy. A robust approach is to encode all varying parameters into the directory or file names, or to store them in a structured metadata file. For example, you can store for each run the beam energy, material type, cut settings, and random seed in a small JSON or text file. This allows you to reconstruct any run later, which is crucial for reproducible research.
When you perform multiple runs with identical setup to reduce statistical uncertainty, the main technical challenge is the combination of results. For scalar quantities such as total counts or integral dose, you can simply sum the relevant quantities over runs. For distributed quantities such as histograms or dose images, you typically add the bin contents from each run to obtain a combined histogram or dose map.
If each run has a different number of events, or if events have different statistical weights, you must use weighted averages to combine results correctly. Suppose you have $n$ independent runs, and for a given quantity $x$ (for example mean dose in a region) you know its estimate $x_i$ and the number of events $N_i$ in run $i$. The combined estimate is the weighted mean:
$$
\bar{x} = \frac{\sum_{i=1}^{n} N_i x_i}{\sum_{i=1}^{n} N_i}.
$$
If instead you keep raw histograms or images, combining runs is typically simpler. You just add the content of corresponding bins across runs. The resulting bin content corresponds to the total counts from all events. From there you can compute normalized quantities, such as counts per primary or dose per unit activity, by dividing by the total number of primaries and other scale factors.
When combining multiple runs:
- Use independent seeds for each run to ensure statistical independence.
- Sum raw quantities such as counts or energy deposition across runs before normalizing.
- Track the number of events or histories in each run, and use this information for correct normalization and uncertainty estimation.
Multiple runs are also essential for uncertainty studies. You might keep the simulation configuration fixed and repeat the simulation several times with different seeds to estimate the spread in a result purely due to Monte Carlo noise. If you record a quantity of interest from each run, such as a dose metric or count rate, you can compute the sample mean and standard deviation over runs. If you have $n$ runs and the values $x_i$, the sample mean and standard deviation are
$$
\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i, \quad
s = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}.
$$
This gives you a direct measure of run-to-run variability, and for large $n$ you can also form confidence intervals for your estimates.
Multiple simulation runs are also useful when varying physical or numerical settings, such as production cuts or physics lists, to study their impact on the results. You can run pairs of simulations that differ in only one parameter, then compare metrics such as dose distributions or energy spectra. This type of parameter comparison is an important part of simulation validation and performance optimization, covered in more detail in other chapters.
To keep overall project complexity under control, you should standardize how you manage multiple runs. Define a clear folder hierarchy, centralize the list of configurations in a single text or table file, and implement a consistent procedure for launching runs and merging results. With this structure in place, scaling from a handful of runs to thousands becomes a matter of increasing job indices rather than redesigning your workflow.
Views: 11