Table of Contents
Measure First, Then Change One Thing
Faster training starts with knowing what is slow. In practice, the bottleneck is usually either the input pipeline, the model compute, or synchronization between CPU and GPU. The most reliable workflow is to change one variable at a time and confirm the effect with timing, because many speed tips interact and some can even slow you down depending on your hardware and batch size.
A simple timing approach is to measure wall clock time per iteration and per epoch after a short warmup, because the first few iterations include one time setup costs like CUDA kernel compilation and caching. When measuring GPU work, remember that CUDA operations are asynchronous, so you only get meaningful timings if you synchronize before stopping the timer. If you forget this, you can mistakenly conclude that a change sped things up when you only changed when the program waits.
Rule: For accurate GPU timing, insert torch.cuda.synchronize() before starting and before stopping the timer. Otherwise timings can be misleading because CUDA launches are asynchronous.
Use the Largest Batch Size That Fits
Batch size is often the biggest lever for throughput because it increases parallel work per step and reduces overhead per sample. The limit is usually GPU memory. A practical approach is to increase batch size until you get an out of memory error, then back off slightly. When you increase batch size, you may need to adjust the learning rate schedule. The details of learning rate tuning belong elsewhere, but the key speed point is that larger batches often reduce total steps per epoch, which reduces overhead from Python and logging.
If memory is the only thing preventing a larger batch, use gradient accumulation to simulate a larger effective batch while keeping micro batches small enough to fit. This increases compute time per optimizer step, but can improve overall utilization and reduce time spent on optimizer overhead and dataloader overhead per sample.
Prefer Mixed Precision on GPUs That Support It
On modern NVIDIA GPUs, mixed precision training can significantly increase throughput and reduce memory use. In PyTorch, this usually means using torch.cuda.amp.autocast for the forward pass and a GradScaler to safely scale gradients. The main speed win comes from using Tensor Cores and reduced memory bandwidth pressure. Mixed precision is not always faster on every model, but it often is for convolutional networks and transformers.
Be careful when comparing runs, because mixed precision can change numerics slightly. If you are seeing instability, verify loss scaling and watch for NaNs, then consider disabling mixed precision temporarily to confirm whether it is the cause.
Rule: Only judge mixed precision speed after a warmup and with correct synchronization timing, because kernel selection and caching can skew first iteration measurements.
Reduce Python Overhead in the Training Step
Python level overhead becomes visible when your model is small or your batch size is small. A few practical habits help.
Keep the per iteration work minimal. Avoid building new tensors in Python loops when you can use vectorized tensor operations. Avoid frequent .item() calls, because they often force synchronization between CPU and GPU. Log less frequently, for example every N steps, and prefer aggregating metrics on GPU and transferring to CPU occasionally.
Set the model to the right mode. Use model.train() during training and model.eval() during evaluation, because layers like dropout and batch norm behave differently and can affect performance and correctness.
If you are on PyTorch 2.x, torch.compile can speed up many models by reducing Python overhead and optimizing graphs, although the payoff depends on the model and the backend. It also adds compile time overhead, so it tends to help more for long training runs than for quick experiments.
Keep the GPU Busy by Feeding Data Efficiently
Even a very fast GPU will sit idle if it waits for batches. When training seems slow and GPU utilization is low, the data pipeline is a common culprit. Use multiple DataLoader workers so preprocessing runs in parallel, and enable pinned memory so host to device transfers are faster. When moving batches to GPU, using non blocking transfers can overlap copies with compute when the rest of the pipeline is configured correctly.
If preprocessing is expensive, consider caching preprocessed data when possible, or moving lightweight transforms onto the GPU. If the dataset is stored on a slow drive, storage can become the bottleneck, so placing data on a fast SSD can make a bigger difference than code tweaks.
Rule: If GPU utilization is low, do not optimize the model first. Fix the input pipeline and host to device transfers so the GPU is consistently fed with data.
Choose Efficient Model and Tensor Operations
Small implementation choices can change performance. Prefer operations that are fused or highly optimized. For example, use built in PyTorch layers and losses rather than manual implementations when possible, because they are typically optimized and may use fused kernels.
Pay attention to tensor memory layout. Many operations are fastest when tensors are contiguous. Excessive transposes and slices can create non contiguous tensors that force implicit copies later. If you see unexpected slowdowns after indexing or permuting, check whether calling .contiguous() at the right point improves performance, but only do this if you have measured that contiguity is the issue, because extra copies can also hurt.
Also minimize unnecessary dtype conversions. Keep data in the dtype you intend to compute with, and avoid repeatedly converting inside the loop.
Avoid Unnecessary Synchronization and Transfers
A common hidden slowdown is moving data between CPU and GPU too often. Keep tensors on the GPU during the training step, and only move results to CPU when needed for logging or saving. Avoid calling .cpu() or .numpy() inside the loop unless necessary, and avoid frequent printing, which can dominate runtime.
When computing metrics, prefer computing them on the GPU and reducing to a scalar occasionally. If you must call .item(), do it sparingly.
Tune Validation and Checkpoint Frequency
Validation and checkpointing are essential, but they can be expensive. If you validate too often, training can appear slow even if the training step is fast. A practical approach is to validate every fixed number of steps or every epoch early on, then less frequently once the run is stable, depending on how quickly the model changes.
Checkpointing can also stall training, especially if you save large optimizer states. Save the minimal state you need at the frequency you need, and consider saving asynchronously in a separate process if you have very strict throughput requirements.
Use Early Stopping for Time to Result
If your goal is to reach a target validation score quickly, the fastest training is often training less. Early stopping based on validation metrics can save substantial time. This is not a compute optimization, but it is a workflow optimization that reduces wasted epochs once the model stops improving.
A Practical Speed Checklist
Start by confirming that the GPU is actually the bottleneck, then increase batch size until memory is full, enable mixed precision if appropriate, and make sure the DataLoader keeps up. Reduce Python overhead by logging less and avoiding synchronizing calls. Only then consider heavier tools like compilation, advanced kernel optimizations, or architectural changes. The consistent theme is to measure, change one thing, and verify the improvement with correct timing.