Table of Contents
When and why gradient accumulation is useful
Gradient accumulation is a technique that lets you train as if you had a larger batch size than your hardware can fit in memory. Instead of computing gradients on one large batch, you split it into several smaller microbatches, run the forward and backward pass on each microbatch, and only update the weights after you have summed gradients across all microbatches. The effect is similar to training with a batch size equal to the microbatch size multiplied by the number of accumulation steps.
This is most useful when your model or input data is large and you hit GPU memory limits, especially with high resolution images, long sequences, or large transformer models. Accumulation trades time for memory, because you do more forward and backward passes per optimizer update.
If you accumulate for $k$ microbatches of size $b$, your effective batch size is $B = k \cdot b$. You should call optimizer.step() once per effective batch, not once per microbatch.
The key rule, scaling the loss correctly
When you backpropagate, the gradient magnitude depends on how the loss is reduced. Many PyTorch losses default to averaging over the batch. If you simply sum gradients across microbatches without adjusting, you will effectively scale your gradients by the number of accumulation steps and change the optimization behavior.
The standard approach is to divide the loss by the number of accumulation steps before calling backward(). That makes the final accumulated gradient match the gradient you would have gotten from one batch containing all samples.
To match large batch training with mean reduced losses, use loss = loss / accum_steps before loss.backward(). This keeps gradient scale consistent when you accumulate.
A minimal PyTorch training pattern
The core pattern is to zero gradients once per effective batch, not every iteration. You backpropagate on each microbatch, then step and zero when you have finished accum_steps microbatches.
In PyTorch, optimizer.zero_grad(set_to_none=True) is commonly used for performance and memory behavior.
A typical structure looks like this in code:
accum_steps = 4
model.train()
optimizer.zero_grad(set_to_none=True)
for step, (x, y) in enumerate(train_loader):
x, y = x.to(device), y.to(device)
y_pred = model(x)
loss = criterion(y_pred, y)
loss = loss / accum_steps
loss.backward()
if (step + 1) % accum_steps == 0:
optimizer.step()
optimizer.zero_grad(set_to_none=True)
If the number of batches is not divisible by accum_steps, you should perform a final optimizer.step() after the loop for the leftover microbatches, otherwise their gradients never get applied.
If the epoch ends with leftover accumulated gradients, you must still call optimizer.step() once, or you silently drop the last updates.
Interaction with gradient clipping
If you use gradient clipping, clip once per optimizer update, after all microbatch gradients have been accumulated and before optimizer.step(). Clipping each microbatch separately is not equivalent to clipping the combined gradient.
A correct placement is:
if (step + 1) % accum_steps == 0:
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
optimizer.step()
optimizer.zero_grad(set_to_none=True)Interaction with mixed precision
With mixed precision training using torch.cuda.amp, you typically use a GradScaler. The accumulation logic stays the same, but you scale the divided loss, and you only unscale and step when you are ready to update.
A common pattern is:
scaler = torch.cuda.amp.GradScaler()
accum_steps = 4
optimizer.zero_grad(set_to_none=True)
for step, (x, y) in enumerate(train_loader):
x, y = x.to(device), y.to(device)
with torch.cuda.amp.autocast():
y_pred = model(x)
loss = criterion(y_pred, y)
loss = loss / accum_steps
scaler.scale(loss).backward()
if (step + 1) % accum_steps == 0:
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad(set_to_none=True)
If you also use gradient clipping with AMP, you generally unscale before clipping. That means calling scaler.unscale_(optimizer) before clip_grad_norm_.
With AMP and clipping, unscale before clipping, then step: scaler.unscale_(optimizer), then clip_grad_norm_, then scaler.step(optimizer).
Learning rate, schedules, and what to consider when changing effective batch size
Gradient accumulation increases the effective batch size, and that can change the training dynamics. If you are using accumulation purely to fit memory while keeping the same effective batch size as before, then you should not change the learning rate. If you are intentionally increasing the effective batch size compared to your previous setup, you may need to revisit your learning rate and learning rate schedule because large batch training can behave differently.
The safest beginner friendly approach is to treat accumulation as a memory trick to reach a desired effective batch size, then tune learning rate normally for that effective batch size, and keep your logging and schedules aligned with optimizer steps rather than microbatch iterations.
Many schedulers are designed around optimizer updates. If you call the scheduler every microbatch while stepping the optimizer every $k$ microbatches, your schedule progresses $k$ times faster than intended.
Logging losses and metrics under accumulation
Because you divide the loss by accum_steps for correct gradients, the value you backpropagate is smaller than the per microbatch loss. For human readable logs, you often want to log the original loss before division, or log the averaged loss over the effective batch.
A simple approach is to keep two values: one for backprop, one for logging. If you log per microbatch, log loss.item() * accum_steps if you already divided, or compute a separate raw_loss before scaling.
Common mistakes and quick checks
A frequent mistake is calling optimizer.zero_grad() every microbatch, which defeats accumulation since it discards previously accumulated gradients. Another is forgetting to divide the loss, which effectively multiplies gradients by accum_steps. A third is stepping the optimizer at the wrong frequency, such as stepping every microbatch while also dividing the loss, which changes the effective learning rate and noise scale in a way that is not equivalent to large batch training.
A quick correctness check is to run one update with a true batch of size $B$ on a small model, then run the same update with microbatches and accumulation, and compare parameter deltas. With deterministic settings and identical data ordering, they should be extremely close when loss scaling and stepping logic are correct.
Correct accumulation requires three aligned choices: do not zero gradients between microbatches, divide loss by accum_steps before backprop, and call optimizer.step() only after the last microbatch in the group.