Table of Contents
Why reproducibility matters
Training a neural network is not like running a normal program where the same input always gives the same output. Many steps in a deep learning workflow contain randomness, and some operations can be nondeterministic even when you control randomness. Reproducibility means that you can re run an experiment and get the same results, or at least results that are meaningfully comparable. This helps you debug, compare models fairly, and trust improvements.
There are two practical levels to aim for. The first is experiment level reproducibility, where runs are similar enough that your conclusions do not depend on lucky randomness. The second is bitwise determinism, where you get identical numbers across runs. Bitwise determinism is harder, sometimes slower, and can still vary across different hardware or library versions. For beginners, you should learn how to control randomness, then decide when strict determinism is worth the trade off.
Where randomness comes from
Randomness appears in parameter initialization, data shuffling, data augmentation, dropout, and some sampling based layers or losses. Even without explicit randomness, some GPU kernels use parallel algorithms where the order of floating point accumulation can vary, leading to tiny numerical differences that can grow over training.
Also note that your results can change if you change any of the following: the random seed, the exact train validation split, the batch size, the hardware type, the PyTorch version, the CUDA version, the cuDNN version, or even the number of DataLoader workers.
Seeds, what they control and what they do not
A seed initializes a pseudorandom number generator. If you set the same seed and the same sequence of random number calls happens in the same order, you will get the same random values. This controls many sources of randomness, but it does not guarantee determinism if the underlying operations are nondeterministic or if your program calls random functions in a different order across runs.
In a typical PyTorch project you may need to seed multiple generators: Python’s built in random, NumPy, and PyTorch. If you use GPUs, you also need to seed CUDA random generators.
Setting a seed is necessary for reproducibility, but it is not sufficient for determinism. You can still see differences because of nondeterministic kernels, multithreading, data loading order, or different hardware and library versions.
A practical seeding recipe for PyTorch
In most beginner projects, a simple function called at the very start of your script or notebook is enough to make runs much more repeatable. You should call it before you create datasets, models, or DataLoaders.
import os
import random
import numpy as np
import torch
def seed_everything(seed: int = 42):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed) # safe even if you do not have CUDA
# Makes some CUDA operations deterministic at the cost of speed.
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
seed_everything(42)The cuDNN settings above reduce variation when training convolutional networks on CUDA. Disabling benchmark prevents cuDNN from picking different convolution algorithms based on timing, which can change between runs.
Deterministic algorithms in PyTorch
PyTorch provides a stricter switch that asks for deterministic implementations where possible and throws an error when an operation is known to be nondeterministic.
torch.use_deterministic_algorithms(True)You can combine this with:
torch.set_float32_matmul_precision("high")if you are on newer PyTorch versions and want more consistent matmul behavior, although performance and exact effects depend on hardware and settings.
On CUDA, some operations require an environment variable for full determinism in certain matrix operations. If you hit an error message that mentions CuBLAS workspace configuration, you can set one of these before starting Python:
For Linux and macOS shells:
export CUBLAS_WORKSPACE_CONFIG=:16:8or
export CUBLAS_WORKSPACE_CONFIG=:4096:8For Windows PowerShell:
setx CUBLAS_WORKSPACE_CONFIG ":16:8"Then restart your terminal or session.
If you enable deterministic algorithms, PyTorch may raise runtime errors for certain layers or ops that do not have deterministic implementations on your device. This is expected. Determinism can reduce performance and sometimes increase memory use.
Reproducibility with DataLoader and multiple workers
The DataLoader can introduce randomness through shuffling and through worker processes. With num_workers greater than 0, each worker has its own random state. To make data order and random transforms repeatable, you should seed workers.
A common pattern is to pass a torch.Generator to the DataLoader for shuffle order, and define worker_init_fn to seed NumPy and Python random inside each worker.
from torch.utils.data import DataLoader
def seed_worker(worker_id):
worker_seed = torch.initial_seed() % 2**32
np.random.seed(worker_seed)
random.seed(worker_seed)
g = torch.Generator()
g.manual_seed(42)
loader = DataLoader(
dataset,
batch_size=64,
shuffle=True,
num_workers=2,
worker_init_fn=seed_worker,
generator=g,
persistent_workers=True
)This makes the shuffling order repeatable and makes random behavior inside each worker consistent. If you use random data augmentation in your dataset’s __getitem__, this matters.
If you want repeatable shuffling, do not rely on global seeding alone. Pass a generator to DataLoader and seed the workers, especially when num_workers is greater than 0.
Determinism versus “similar results”
Even after doing everything above, strict equality is not always realistic across different GPUs or across CPU versus GPU, because floating point arithmetic can differ. A good habit is to decide what you need.
If you are debugging, aim for determinism on one machine. If you are comparing two model ideas, aim for fair comparison by keeping the seed, split, and training settings fixed, then repeat across multiple seeds to make sure the improvement is not luck.
A simple way to report robustness is to run the same experiment for several seeds and report mean and standard deviation of your metric. The number of seeds depends on time, but even 3 to 5 seeds is often more informative than one perfect run.
What to record so you can reproduce later
Reproducibility is also about being able to reconstruct the exact conditions. In practice you should save the seed, the full set of hyperparameters, the dataset version or checksum, and the exact package versions.
At minimum, record your PyTorch version and whether you used CUDA:
import torch
print(torch.__version__)
print(torch.version.cuda)
print(torch.backends.cudnn.version())
print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU")If you later load a checkpoint and results do not match, mismatched library versions and nondeterministic kernels are common causes.
A minimal checklist
Set all relevant seeds at the very start. Control DataLoader shuffling and worker seeds. Decide whether to enable deterministic algorithms based on your goal. Record versions, hardware, and all training settings. When comparing approaches, run multiple seeds and compare averages, not just a single run.
Always fix the train validation split when you compare models. Changing the split can easily change metrics more than most modeling tweaks, and it can make improvements look real when they are not.