41.3. Job Scripts

Table of Contents

Running Python simulations

On a high performance computing system you usually do not run GATE interactively. Instead, you submit a batch job, and the scheduler (for example Slurm) decides when and where your Python simulation runs. A job script is simply a text file that contains two things: scheduler directives at the top, and the shell commands that actually run your Python script.

Even for beginners it is helpful to think of the job script as a “reproducible terminal session.” Everything you would normally type in a terminal to run your GATE simulation should be placed inside this script, in the correct order. You then submit the script with a command such as sbatch job.sh instead of typing the commands manually.

A minimal job script for a Python based GATE simulation often looks like a regular shell script with some extra lines at the top for the scheduler. A typical layout in Bash is:

bash

#!/bin/bash
# Scheduler options (for example, Slurm directives) go here
# 1. Load the software environment
# 2. Move to the run directory
# 3. Run the Python simulation
python my_gate_simulation.py

In practice you will add more detail. You usually specify the job name, where to save log files, how many CPU cores you need, and how long the job is allowed to run. For example, on a Slurm cluster, the top of the script might include lines starting with #SBATCH that describe your resource requirements and logging preferences.

It is good practice to run your Python script from a dedicated run directory that contains your configuration files and that will hold the output files. Inside the job script, you can change into this directory with cd /path/to/run_dir before executing Python. This keeps paths consistent and simplifies the collection of results after the job finishes.

You should also always capture the job’s textual output. The scheduler can write both standard output and error messages into files. These files are essential for debugging failed jobs and for keeping a trace of what happened during the simulation. In your job script you may additionally redirect output explicitly with:

bash

python my_gate_simulation.py > run.log 2>&1

This directs normal output to run.log and also redirects error messages into the same file. Some schedulers already provide their own log files, but having a named log in your run directory is often more convenient.

When using multithreading in GATE, your Python script will usually configure the number of worker threads. You must then choose matching resource requests in the job script. For example, if the simulation uses 8 threads internally, you should request 8 CPU cores from the scheduler and not just 1. Forgetting this leads to oversubscription and slower performance.

If you plan to run the same simulation with different parameters, for example different random seeds, you can write a generic job script that reads these parameters from environment variables or from a configuration file. The scheduler can then submit many instances of the same script, each with its own seed or configuration. The logic inside your Python simulation can read these values, so you keep only a single, well maintained job template.

A job script must be non interactive. Do not use commands that wait for user input, graphical windows, or interactive prompts. Any pause or visualization request will usually cause the job to hang until the scheduler kills it.

Environment setup

Inside an HPC job script, you cannot rely on your usual interactive environment. Environment modules, virtual environments, and paths must be configured explicitly before you call Python. The key tasks are to select the correct Python interpreter, ensure that Geant4 and OpenGATE are available, and define any required environment variables such as G4DATA paths.

Many clusters use environment modules. At the beginning of the script you typically load the module that provides Python and sometimes a module that provides a complete GATE or Geant4 installation. For example:

bash

module purge
module load python/3.10
module load geant4
module load gate

If your GATE installation is in a Python virtual environment or Conda environment, you activate that environment in the script. This must be done after any required modules are loaded, because the environment depends on the correct system libraries. For a virtual environment you might use:

bash

source /path/to/venv/bin/activate

For Conda, a typical sequence is:

bash

source /path/to/miniconda3/etc/profile.d/conda.sh
conda activate gate_env

The important detail is that activation is done inside the job script itself. Activating the environment in an interactive login session is not sufficient, because batch jobs run in a fresh, non interactive shell.

If your cluster does not provide Geant4 data paths automatically, you must set them yourself. That usually means exporting environment variables that point to the data directories, for example:

bash

export G4LEDATA=/path/to/geant4/data/G4EMLOW
export G4LEVELGAMMADATA=/path/to/geant4/data/PhotonEvaporation

These paths are the same variables you would set when running GATE on your own workstation. The difference on an HPC system is that they must be present inside the job script, so that each compute node receives a complete environment.

You should also set a working directory, especially if the scheduler starts your job from your home directory by default. Placing a cd command near the top of the script ensures that all relative paths in your Python code are interpreted correctly:

bash

cd /path/to/project/run_dir

This directory can contain configuration files, input images such as CT data, and a subdirectory for simulation output. Your Python simulation can then use simple relative paths like ./output or ./input/images.

To help with reproducibility, many users record environment information in the job output. You can, for instance, print the active Python version, the GATE version, and the list of loaded modules at the beginning of each run. A short block like:

bash

python -c "import sys, opengate; print(sys.version); print(opengate.__version__)"

writes version information into the log file and makes it easier to understand differences between simulations.

Finally, if you use random seeds for reproducibility, you often define them in the environment as well. The scheduler can pass different seed values to each job instance, and your Python code can read them from os.environ. This pattern keeps the simulation code stable while the environment provides run specific configuration.

Always configure Python, GATE, and all required environment variables inside the job script itself. Never assume that the environment of your interactive login session is automatically available on compute nodes.

41.3. Job Scripts

Running Python simulations

Environment setup

Comments

Where to Move