Table of Contents
What Performance and Scaling Means in Practice
Performance in deep learning usually means how fast you can train or run inference, how efficiently you use CPU and GPU resources, and how reliably you can fit a model and batch size into memory. Scaling means what changes when you try to do more, such as using larger models, larger batches, more data, or more hardware. In PyTorch, most performance work is about keeping the GPU busy, avoiding unnecessary CPU work, moving less data between CPU and GPU, and preventing memory spikes that force smaller batch sizes or cause out of memory errors.
This part of the course focuses on practical, beginner friendly habits: measuring before guessing, understanding where time is spent, and applying a small set of high impact techniques. Each technique will be covered in detail in its own chapter later in this section, so here the goal is to build the mental map of what matters and when to reach for which tool.
The Main Bottlenecks You Will Encounter
Training time is usually limited by one of three things. The first is compute, meaning your model’s forward and backward pass dominate. The second is input pipeline speed, meaning the GPU waits for the next batch because the CPU is loading and preprocessing too slowly. The third is memory, meaning you cannot use the batch size you want, you trigger frequent allocations, or you hit out of memory and crash.
A helpful way to think about a training step is as a pipeline: the DataLoader prepares a batch on the CPU, the batch is transferred to the device, the model computes outputs and loss, gradients are computed, and the optimizer updates parameters. If any stage is slow, it becomes the limiting stage. Optimizing a stage that is not limiting will not help much.
Rule: Do not optimize blindly. Measure step time and identify whether you are compute bound, input bound, or memory bound before changing code.
Measuring Speed Without Fooling Yourself
Beginners often time code in a way that produces misleading results. On GPUs, many operations are asynchronous, which means Python may continue before the GPU finishes. If you measure time without synchronizing, you may see unrealistically small durations. Proper benchmarking also requires a warmup period because the first few iterations often include one time setup costs, memory allocations, and kernel compilation effects.
For quick checks, you can measure iteration time over many steps and compute an average. When using CUDA, synchronize right before stopping the timer so you measure real GPU completion time. You will also want to separate data loading time from compute time by timing the DataLoader iteration independently from the forward and backward pass timing.
Rule: When benchmarking CUDA code, include synchronization around timing, or your numbers can be wrong.
The Two Levers That Usually Matter Most
For most beginner projects, the biggest wins typically come from improving the data pipeline and improving GPU utilization. Data pipeline improvements include choosing reasonable num_workers, enabling pinned memory when transferring to GPU, and avoiding expensive Python level transforms per sample. GPU utilization improvements include using larger batch sizes when memory allows, reducing unnecessary device transfers, and using mixed precision where appropriate.
Memory is the constraint that often blocks these improvements. If you cannot increase batch size, you may need memory saving techniques like gradient accumulation or checkpointing. Those topics are addressed later in this section at an overview level or in dedicated chapters.
Data Movement Costs and Why They Hurt
Copying batches from CPU to GPU has overhead. Small, frequent copies are worse than fewer, larger copies. Copying tensors repeatedly inside the model forward pass is especially harmful. A common performance mistake is creating new tensors on the CPU inside forward and then moving them to the GPU each step. Another is calling .to(device) on the same tensors multiple times.
The simple principle is that data should arrive on the correct device once per batch, and then stay there during the step. Model parameters should also live on the target device, and you should avoid implicit device mismatches that force hidden copies or errors.
Rule: Move the model to the device once, and move each batch to the device once. Avoid repeated .to(device) calls inside forward.
Memory: The Constraint Behind Batch Size and Stability
GPU memory usage during training is not just model weights. The dominant term is often activation memory stored for backpropagation. Larger batch sizes increase activation memory roughly linearly. Some layers and operations also have higher memory footprints.
A simple mental model is that training memory is composed of parameters, gradients, optimizer state, and saved activations. Optimizers like Adam keep extra state tensors, which increases memory compared to SGD. This is one reason a model that fits with SGD may not fit with Adam at the same batch size.
When you see out of memory errors, you should think in terms of reducing peak memory, not just total memory. A single large activation or a temporary tensor can push peak usage over the limit.
Rule: Out of memory is about peak memory. One large temporary tensor can crash training even if average usage looks fine.
Scaling Up: What Changes As You Grow Workloads
As models and datasets grow, the cost structure changes. Larger models tend to become compute bound on the GPU, which shifts attention toward GPU kernels, mixed precision, and efficient architectures. Larger datasets often increase the importance of I, O and preprocessing, which shifts attention toward DataLoader settings, caching, and minimizing per sample Python work.
When you scale batch size, learning dynamics can change, even if performance improves. Larger batches reduce the number of optimizer steps per epoch, so you often adjust learning rate schedules and sometimes learning rates themselves. Those training dynamics details are covered elsewhere in the course, but it is important to remember that speed optimizations can change training behavior if they change effective batch size or numerical precision.
The Practical Workflow for Performance Work
A good workflow is iterative. First, establish a baseline run with consistent settings and record throughput, such as examples per second, and memory headroom. Second, change one thing at a time, such as DataLoader workers or mixed precision, and re measure. Third, keep changes that improve the metric you care about without breaking training stability or accuracy.
You should also decide what success means for your project. Sometimes the goal is to maximize throughput. Sometimes it is to fit a model into memory. Sometimes it is to reduce iteration time enough that experimentation becomes easier. Different goals lead to different choices.
How This Section Fits Into the Rest of the Course
This section is a toolbox. You will learn how to find bottlenecks, speed up your input pipeline, use the GPU efficiently, manage memory, and understand the basic ideas behind distributed training. The emphasis is not on exotic tricks but on reliable techniques you will use repeatedly.
The chapters that follow will cover profiling, DataLoader optimization, GPU utilization and memory management, gradient accumulation, distributed training concepts, and practical guidelines for faster training.