40.6. Large Simulations
Table of Contents
Batch processing
When you move from small test cases to realistic medical physics problems in GATE, the number of events, the size of the geometry, and the complexity of the physics can easily push single simulations to hours or days of CPU time. At that point, you rarely run one huge job. Instead, you split the work into many smaller jobs that can run in parallel on a workstation or HPC system. This systematic splitting and automated execution is what we call batch processing.
In a batch workflow you do not start simulations interactively, and you do not rely on visualization. You prepare scripts and configuration files that completely define the simulation, then submit many jobs that execute without user interaction and write results to well-defined output directories.
A key ingredient of batch processing is a clear separation between your “simulation definition” and your “run configuration.” The simulation definition describes geometry, materials, physics, sources, actors, and digitizers. It should be as independent as possible from details like the number of events, random seed, or the specific output path. Those run-time details are then controlled by external parameters, for example environment variables, command line arguments, or small configuration files.
You can implement this in Python by reading parameters at the top of your script. For instance, you might read the number of events and a seed from the command line, then pass them into your simulation object and into your random number configuration. This way, the same script can be reused thousands of times with different configurations during batch processing.
A typical batch strategy for large simulations is to partition your total number of events into many jobs. Each job runs the same geometry and physics, but with a smaller event count and a different random seed. For example, if you need $10^9$ events, you might run 100 jobs with $10^7$ events each. This improves resilience because if one job crashes, you only lose a fraction of the statistics, and it allows the use of many cores or nodes in parallel.
Another important aspect of batch processing is output organization. For large simulations you will produce a significant number of files, so you should adopt a consistent directory structure and naming scheme. An effective pattern is to create one directory per job that contains all outputs for that job, along with a small metadata file describing configuration parameters such as the seed, number of events, and software version. You can encode the job index or seed into the folder name so that you can later reconstruct how each file was produced.
Use of logging also becomes crucial in batch processing. Since you do not see the terminal during execution, you should redirect standard output and error logs to files. You can configure GATE or your Python script to write a log per job, containing key information such as simulation start and end time, number of events processed, and any warnings. These logs are essential for debugging problems that arise only at large scale.
When you run many jobs concurrently, I/O and file size can become a bottleneck. To keep batch processing efficient, you should limit the amount of data per event. Only record what you truly need for your analysis. For instance, instead of writing full hits, singles, and coincidences for every event in a huge PET study, you might only record singles and a few summary actors, or use phase space data efficiently.
In large batch simulations, always:
- Run a small test with a tiny number of events and the exact same script and configuration.
- Verify output content and size before scaling to many jobs.
- Control random seeds explicitly so jobs are independent and reproducible.
Finally, batch processing is closely related to using HPC schedulers like Slurm, which are covered elsewhere. Regardless of the platform, the core ideas remain the same: noninteractive runs, parameterized scripts, controlled random seeds, and disciplined output management.
Multiple simulation runs
Large projects rarely rely on a single simulation result. Instead, you typically perform multiple simulation runs that explore different conditions, verify convergence, or provide uncertainty estimates.
One common reason for multiple runs is to improve statistical precision by combining independent simulations. Since Monte Carlo uncertainty scales roughly as $1/\sqrt{N}$, you can achieve the same total number of events by summing many smaller runs. In practical terms, you can run $K$ independent simulations, each with $N$ events, and then merge the outputs as if you had run one simulation with $K \times N$ events, provided that the random seeds and histories are independent and the physics and geometry are identical.
Another purpose of multiple runs is systematic parameter studies. You may want to scan over beam energies, source activities, detector configurations, or geometry dimensions. For each set of parameters, you run a separate simulation and compare the output. In such studies, you typically define a parameter grid, then generate one configuration for each point. With Python, you can automate this by looping over parameter values and writing distinct configuration files or passing parameters directly on the command line when launching each run.
To keep track of many runs, you need a clear naming and indexing strategy. A robust approach is to encode all varying parameters into the directory or file names, or to store them in a structured metadata file. For example, you can store for each run the beam energy, material type, cut settings, and random seed in a small JSON or text file. This allows you to reconstruct any run later, which is crucial for reproducible research.
When you perform multiple runs with identical setup to reduce statistical uncertainty, the main technical challenge is the combination of results. For scalar quantities such as total counts or integral dose, you can simply sum the relevant quantities over runs. For distributed quantities such as histograms or dose images, you typically add the bin contents from each run to obtain a combined histogram or dose map.
If each run has a different number of events, or if events have different statistical weights, you must use weighted averages to combine results correctly. Suppose you have $n$ independent runs, and for a given quantity $x$ (for example mean dose in a region) you know its estimate $x_i$ and the number of events $N_i$ in run $i$. The combined estimate is the weighted mean:
$$
\bar{x} = \frac{\sum_{i=1}^{n} N_i x_i}{\sum_{i=1}^{n} N_i}.
$$
If instead you keep raw histograms or images, combining runs is typically simpler. You just add the content of corresponding bins across runs. The resulting bin content corresponds to the total counts from all events. From there you can compute normalized quantities, such as counts per primary or dose per unit activity, by dividing by the total number of primaries and other scale factors.
When combining multiple runs:
- Use independent seeds for each run to ensure statistical independence.
- Sum raw quantities such as counts or energy deposition across runs before normalizing.
- Track the number of events or histories in each run, and use this information for correct normalization and uncertainty estimation.
Multiple runs are also essential for uncertainty studies. You might keep the simulation configuration fixed and repeat the simulation several times with different seeds to estimate the spread in a result purely due to Monte Carlo noise. If you record a quantity of interest from each run, such as a dose metric or count rate, you can compute the sample mean and standard deviation over runs. If you have $n$ runs and the values $x_i$, the sample mean and standard deviation are
$$
\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i, \quad
s = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}.
$$
This gives you a direct measure of run-to-run variability, and for large $n$ you can also form confidence intervals for your estimates.
Multiple simulation runs are also useful when varying physical or numerical settings, such as production cuts or physics lists, to study their impact on the results. You can run pairs of simulations that differ in only one parameter, then compare metrics such as dose distributions or energy spectra. This type of parameter comparison is an important part of simulation validation and performance optimization, covered in more detail in other chapters.
To keep overall project complexity under control, you should standardize how you manage multiple runs. Define a clear folder hierarchy, centralize the list of configurations in a single text or table file, and implement a consistent procedure for launching runs and merging results. With this structure in place, scaling from a handful of runs to thousands becomes a matter of increasing job indices rather than redesigning your workflow.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Introduction to GATE
- ☰ 1.1. What Is GATE?
- ☰ 1.2. Applications of GATE
- ☰ 1.3. GATE vs Geant4
- ☰ 1.4. GATE Simulation Workflow
-
- ☰ 2. Installing GATE
- ☰ 2.1. GATE Requirements
- ☰ 2.2. Creating a Python Environment
- ☰ 2.3. Installing GATE
- ☰ 2.4. Verifying the Installation
- ☰ 2.5. GATE Example Simulations
-
- ☰ 3. Python Basics for GATE
- ☰ 3.1. Variables and Data Types
- ☰ 3.2. Conditional Statements
- ☰ 3.3. Loops
- ☰ 3.4. Functions
- ☰ 3.5. Python Modules
- ☰ 3.6. NumPy Basics
- ☰ 3.7. Working with Files
-
- ☰ 4. Creating Your First GATE Simulation
- ☰ 4.1. Importing GATE
- ☰ 4.2. Creating a Simulation
- ☰ 4.3. Adding the World
- ☰ 4.4. Adding a Particle Source
- ☰ 4.5. Adding an Actor
- ☰ 4.6. Running the Simulation
- ☰ 4.7. Inspecting the Results
-
- ☰ 5. Units in GATE
- ☰ 5.1. Physical Units
- ☰ 5.2. Length Units
- ☰ 5.3. Energy Units
- ☰ 5.4. Time Units
- ☰ 5.5. Radioactivity Units
- ☰ 5.6. Using Units Correctly
-
- ☰ 6. Geometry Fundamentals
- ☰ 6.1. The World Volume
- ☰ 6.2. Creating Volumes
- ☰ 6.3. Box Geometry
- ☰ 6.4. Cylindrical Geometry
- ☰ 6.5. Spherical Geometry
- ☰ 6.6. Position and Translation
- ☰ 6.7. Rotation
-
- ☰ 7. Building Complex Geometry
- ☰ 7.1. Nested Volumes
- ☰ 7.2. Repeated Volumes
- ☰ 7.3. Ring Geometry
- ☰ 7.4. Boolean Geometry
- ☰ 7.5. Geometry Overlap
- ☰ 7.6. Reusable Geometry Functions
-
- ☰ 8. Materials
- ☰ 8.1. Material Definitions
- ☰ 8.2. Geant4 Material Database
- ☰ 8.3. Common Medical Materials
- ☰ 8.4. Detector Materials
- ☰ 8.5. Shielding Materials
- ☰ 8.6. Creating Custom Materials
-
- ☰ 9. Geometry Visualization
- ☰ 9.1. Enabling Visualization
- ☰ 9.2. Displaying Volumes
- ☰ 9.3. Changing the View
- ☰ 9.4. Volume Appearance
- ☰ 9.5. Visualizing Particle Tracks
- ☰ 9.6. Debugging with Visualization
-
- ☰ 10. Particle Sources
- ☰ 10.1. Generic Sources
- ☰ 10.2. Particle Types
- ☰ 10.3. Source Position
- ☰ 10.4. Source Direction
- ☰ 10.5. Source Energy
- ☰ 10.6. Source Activity
-
- ☰ 11. Radioactive Sources
- ☰ 11.1. Radioactive Decay
- ☰ 11.2. Medical Radionuclides
- ☰ 11.3. Positron Emitters
- ☰ 11.4. Gamma Emitters
- ☰ 11.5. Activity Distribution
- ☰ 11.6. Time-Dependent Activity
-
- ☰ 12. Physics Lists
- ☰ 12.1. What Is a Physics List?
- ☰ 12.2. Electromagnetic Physics
- ☰ 12.3. Gamma Physics
- ☰ 12.4. Electron Physics
- ☰ 12.5. Positron Physics
- ☰ 12.6. Hadronic Physics
- ☰ 12.7. Choosing Physics Models
-
- ☰ 13. Production Cuts and Transport Parameters
- ☰ 13.1. What Are Production Cuts?
- ☰ 13.2. Range Cuts
- ☰ 13.3. Region-Based Cuts
- ☰ 13.4. Accuracy vs Performance
-
- ☰ 14. Actors
- ☰ 14.1. What Is an Actor?
- ☰ 14.2. Simulation Statistics Actor
- ☰ 14.3. Energy Deposition Actor
- ☰ 14.4. Dose Actor
- ☰ 14.5. Phase Space Actor
- ☰ 14.6. Particle Fluence
- ☰ 14.7. Actor Filters
-
- ☰ 15. Dose Calculation
- ☰ 15.1. Energy Deposition
- ☰ 15.2. Absorbed Dose
- ☰ 15.3. Creating a Dose Actor
- ☰ 15.4. Dose Images
- ☰ 15.5. Statistical Uncertainty
- ☰ 15.6. Dose Profiles
-
- ☰ 16. Hits and Detector Response
- ☰ 16.1. Particle Interactions in Detectors
- ☰ 16.2. Hits
- ☰ 16.3. Hit Collections
- ☰ 16.4. Hit Attributes
- ☰ 16.5. Hits vs Detector Signals
-
- ☰ 17. Digitizers
- ☰ 17.1. What Is Digitization?
- ☰ 17.2. Digitizer Chain
- ☰ 17.3. Energy Summation
- ☰ 17.4. Readout
- ☰ 17.5. Energy Blurring
- ☰ 17.6. Spatial Blurring
- ☰ 17.7. Energy Thresholds
- ☰ 17.8. Time Blurring
-
- ☰ 18. Singles
- ☰ 18.1. What Are Singles?
- ☰ 18.2. Creating Singles
- ☰ 18.3. Detector Identification
- ☰ 18.4. Energy Windows
- ☰ 18.5. Analyzing Singles
-
- ☰ 19. Coincidences
- ☰ 19.1. What Are Coincidences?
- ☰ 19.2. Coincidence Sorting
- ☰ 19.3. True Coincidences
- ☰ 19.4. Scattered Coincidences
- ☰ 19.5. Random Coincidences
- ☰ 19.6. Coincidence Timing
- ☰ 19.7. Coincidence Output
-
- ☰ 20. Data Output
- ☰ 20.1. Simulation Output
- ☰ 20.2. ROOT Output
- ☰ 20.3. Image Output
- ☰ 20.4. Text-Based Output
- ☰ 20.5. Organizing Simulation Results
-
- ☰ 21. GATE and ROOT
- ☰ 21.1. Reading GATE ROOT Files
- ☰ 21.2. Inspecting Output
- ☰ 21.3. Energy Histograms
- ☰ 21.4. Time Histograms
- ☰ 21.5. Detector Maps
- ☰ 21.6. Filtering Events
-
- ☰ 22. GATE and Python Data Analysis
- ☰ 22.1. Reading Simulation Output with Python
- ☰ 22.2. Energy Spectra
- ☰ 22.3. Position Distributions
- ☰ 22.4. Timing Analysis
- ☰ 22.5. Statistical Analysis
- ☰ 22.6. Visualization
-
- ☰ 23. Voxelized Geometry
- ☰ 23.1. Introduction to Voxelized Geometry
- ☰ 23.2. Image-Based Volumes
- ☰ 23.3. Material Mapping
- ☰ 23.4. CT-to-Material Conversion
- ☰ 23.5. Patient Geometry
- ☰ 23.6. Visualizing Voxelized Geometry
-
- ☰ 24. Image and DICOM Data
- ☰ 24.1. Medical Image Formats
- ☰ 24.2. Reading CT Images
- ☰ 24.3. DICOM CT Data
- ☰ 24.4. RT Structure Sets
- ☰ 24.5. RT Plans
- ☰ 24.6. Coordinate Systems
-
- ☰ 25. Motion and Time-Dependent Simulations
- ☰ 25.1. Time in GATE
- ☰ 25.2. Moving Sources
- ☰ 25.3. Moving Detectors
- ☰ 25.4. Respiratory Motion
- ☰ 25.5. Scanner Rotation
- ☰ 25.6. Dynamic Imaging
-
- ☰ 26. PET Simulation Fundamentals
- ☰ 26.1. PET Physics
- ☰ 26.2. PET Scanner Components
- ☰ 26.3. PET Detector Geometry
- ☰ 26.4. PET Source
- ☰ 26.5. PET Physics Configuration
- ☰ 26.6. PET Detector Response
-
- ☰ 27. Building a PET Scanner
- ☰ 27.1. Creating the Detector Ring
- ☰ 27.2. Creating Detector Blocks
- ☰ 27.3. Creating Detector Crystals
- ☰ 27.4. Assigning Detector IDs
- ☰ 27.5. Adding a PET Source
- ☰ 27.6. Configuring the Digitizer
- ☰ 27.7. Running the PET Simulation
-
- ☰ 28. PET Coincidence Analysis
- ☰ 28.1. Coincidence Events
- ☰ 28.2. Energy Selection
- ☰ 28.3. Coincidence Time Window
- ☰ 28.4. Lines of Response
- ☰ 28.5. True Events
- ☰ 28.6. Scatter Events
- ☰ 28.7. Random Events
- ☰ 28.8. PET Performance Metrics
-
- ☰ 29. Time-of-Flight PET
- ☰ 29.1. TOF PET Principles
- ☰ 29.2. Detector Timing Resolution
- ☰ 29.3. Time Difference
- ☰ 29.4. Timing Blurring
- ☰ 29.5. Analyzing TOF Events
-
- ☰ 30. SPECT Simulation Fundamentals
- ☰ 30.1. SPECT Physics
- ☰ 30.2. SPECT Scanner Components
- ☰ 30.3. SPECT Radionuclides
- ☰ 30.4. Gamma Camera Geometry
- ☰ 30.5. SPECT Source
-
- ☰ 31. Building a Gamma Camera
- ☰ 31.1. Detector Head
- ☰ 31.2. Collimator
- ☰ 31.3. Shielding
- ☰ 31.4. Energy Resolution
- ☰ 31.5. Energy Window
- ☰ 31.6. Projection Data
-
- ☰ 32. SPECT Acquisition
- ☰ 32.1. Rotating Gamma Camera
- ☰ 32.2. Projection Angles
- ☰ 32.3. Acquisition Time
- ☰ 32.4. Energy Windows
- ☰ 32.5. Projection Images
- ☰ 32.6. SPECT Data Analysis
-
- ☰ 33. CT and X-Ray Simulation
- ☰ 33.1. X-Ray Physics
- ☰ 33.2. X-Ray Source
- ☰ 33.3. CT Geometry
- ☰ 33.4. Detector Array
- ☰ 33.5. Rotating CT Geometry
- ☰ 33.6. CT Projection Data
-
- ☰ 34. Radiation Therapy Simulation
- ☰ 34.1. External Beam Radiotherapy
- ☰ 34.2. Photon Beams
- ☰ 34.3. Electron Beams
- ☰ 34.4. Dose Calculation
- ☰ 34.5. Beam Geometry
- ☰ 34.6. Dose Validation
-
- ☰ 35. Proton Therapy Simulation
- ☰ 35.1. Proton Physics
- ☰ 35.2. Proton Beam
- ☰ 35.3. Water Phantom
- ☰ 35.4. Bragg Peak
- ☰ 35.5. Spread-Out Bragg Peak
- ☰ 35.6. Proton Dose Analysis
-
- ☰ 36. Internal Dosimetry
- ☰ 36.1. Radioactive Sources in the Body
- ☰ 36.2. Voxelized Activity Maps
- ☰ 36.3. Energy Deposition
- ☰ 36.4. Dose Maps
- ☰ 36.5. Radionuclide Therapy
- ☰ 36.6. Dose Analysis
-
- ☰ 37. Optical Photon Simulation
- ☰ 37.1. Optical Physics
- ☰ 37.2. Scintillation
- ☰ 37.3. Optical Properties
- ☰ 37.4. Optical Surfaces
- ☰ 37.5. Photodetectors
- ☰ 37.6. Optical Photon Detection
-
- ☰ 38. Phase-Space Data
- ☰ 38.1. What Is a Phase Space?
- ☰ 38.2. Recording Phase-Space Data
- ☰ 38.3. Phase-Space Files
- ☰ 38.4. Phase-Space Sources
- ☰ 38.5. Applications
-
- ☰ 39. Random Numbers and Reproducibility
- ☰ 39.1. Monte Carlo Random Numbers
- ☰ 39.2. Random Seeds
- ☰ 39.3. Independent Simulations
- ☰ 39.4. Reproducible Research
-
- ☰ 40. Multithreading and Performance
- ☰ 40.1. Multithreaded Simulation
- ☰ 40.2. Selecting the Number of Threads
- ☰ 40.3. Simulation Speed
- ☰ 40.4. Optimizing Geometry
- ☰ 40.5. Optimizing Output
- ☰ 40.6. Large Simulations
-
- ☰ 41. Running GATE on HPC Systems
- ☰ 41.1. Why Use HPC?
- ☰ 41.2. Running GATE Without Visualization
- ☰ 41.3. Job Scripts
- ☰ 41.4. Running with Slurm
- ☰ 41.5. Running Multiple Simulations
- ☰ 41.6. Combining Results
-
- ☰ 42. Simulation Validation
- ☰ 42.1. Why Validate a GATE Simulation?
- ☰ 42.2. Geometry Validation
- ☰ 42.3. Physics Validation
- ☰ 42.4. Comparing with Analytical Models
- ☰ 42.5. Comparing with Experimental Data
- ☰ 42.6. Statistical Validation
-
- ☰ 43. Debugging GATE Simulations
- ☰ 43.1. Python Errors
- ☰ 43.2. Geometry Problems
- ☰ 43.3. Material Errors
- ☰ 43.4. Source Problems
- ☰ 43.5. Physics Problems
- ☰ 43.6. Actor Problems
- ☰ 43.7. Digitizer Problems
- ☰ 43.8. Performance Problems
-
- ☰ 44. Writing Better GATE Simulations
- ☰ 44.1. Organizing Simulation Scripts
- ☰ 44.2. Configuration Files
- ☰ 44.3. Avoiding Hard-Coded Values
- ☰ 44.4. Reusable Components
- ☰ 44.5. Output Organization
- ☰ 44.6. Documentation
- ☰ 44.7. Version Control
-
- ☰ 45. Practical Example: Gamma-Ray Detector
- ☰ 45.1. Project Overview
- ☰ 45.2. Creating the World
- ☰ 45.3. Creating the Detector
- ☰ 45.4. Creating a Gamma Source
- ☰ 45.5. Configuring Electromagnetic Physics
- ☰ 45.6. Recording Energy Deposition
- ☰ 45.7. Adding Energy Resolution
- ☰ 45.8. Creating an Energy Spectrum
- ☰ 45.9. Analyzing the Spectrum
-
- ☰ 46. Practical Example: Radiation Shielding
- ☰ 46.1. Project Overview
- ☰ 46.2. Creating the Source
- ☰ 46.3. Creating the Shield
- ☰ 46.4. Selecting Shielding Materials
- ☰ 46.5. Changing Shield Thickness
- ☰ 46.6. Recording Transmitted Photons
- ☰ 46.7. Calculating Transmission
- ☰ 46.8. Calculating Attenuation
- ☰ 46.9. Comparing with the Exponential Attenuation Law
-
- ☰ 47. Practical Example: PET Scanner
- ☰ 47.1. Project Overview
- ☰ 47.2. Creating the PET Ring
- ☰ 47.3. Creating Detector Modules
- ☰ 47.4. Creating Detector Crystals
- ☰ 47.5. Adding an F-18 Source
- ☰ 47.6. Configuring PET Physics
- ☰ 47.7. Recording Hits
- ☰ 47.8. Creating Singles
- ☰ 47.9. Applying Energy Blurring
- ☰ 47.10. Applying the PET Energy Window
- ☰ 47.11. Sorting Coincidences
- ☰ 47.12. Identifying True Coincidences
- ☰ 47.13. Identifying Scattered Coincidences
- ☰ 47.14. Creating Lines of Response
- ☰ 47.15. Saving PET Data
-
Comments
Please login to add a comment.
Don't have an account? Register now!