Table of Contents
What Regularization Is and Why You Need It
Regularization is anything you add to training that makes a model less likely to memorize the training set and more likely to generalize to new data. In practice, regularization usually increases training loss a bit while reducing validation loss and improving real world performance. You will notice regularization matters most when the model has enough capacity to overfit, for example many parameters relative to the dataset size, high input dimensionality, noisy labels, or too many training epochs.
A useful mental model is that you are optimizing not just the data fitting term, but a slightly modified objective that discourages overly complex solutions. For many common methods, this looks like
$$
\mathcal{L}_{total}(\theta) = \mathcal{L}_{data}(\theta) + \lambda \, \Omega(\theta)
$$
where $\theta$ are the parameters, $\Omega(\theta)$ is a penalty that grows when the model gets too complex, and $\lambda$ controls how strong the penalty is.
Regularization is not a substitute for a validation set. You still need validation to choose regularization strength and to detect overfitting.
L2 Regularization, Weight Decay in Optimizers
The most common regularization for neural networks is L2 regularization on weights, which penalizes large weights via $\Omega(\theta) = \lVert \theta \rVert_2^2$. The key practical concept in PyTorch is that you typically apply it through the optimizer as weight decay.
With L2 regularization, the objective becomes:
$$
\mathcal{L}_{total} = \mathcal{L}_{data} + \lambda \sum_i \theta_i^2
$$
In PyTorch, many optimizers accept a weight_decay argument. For example, with SGD:
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=1e-4)
For Adam type optimizers, it is usually better to use AdamW, which implements decoupled weight decay:
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)Decoupled weight decay tends to behave more predictably because it applies the decay separately from the adaptive gradient update.
Do not assume weight_decay is always identical to classic L2 regularization for every optimizer. If you use Adam, prefer AdamW when you want weight decay regularization.
What Not to Decay, Biases and Normalization Parameters
In many models, you should avoid applying weight decay to bias terms and to parameters of normalization layers. The reason is that these parameters do not control complexity in the same way as weight matrices, and decaying them can sometimes hurt optimization or accuracy.
A typical approach is to create parameter groups, one with decay and one without:
decay, no_decay = [], []
for name, param in model.named_parameters():
if not param.requires_grad:
continue
if name.endswith(".bias") or "bn" in name.lower() or "norm" in name.lower():
no_decay.append(param)
else:
decay.append(param)
optimizer = torch.optim.AdamW(
[{"params": decay, "weight_decay": 1e-2},
{"params": no_decay, "weight_decay": 0.0}],
lr=3e-4
)This pattern becomes more important as you move to modern architectures that use many normalization layers.
If you apply weight decay to everything by default, you may unintentionally regularize biases and normalization parameters, which can reduce performance.
Dropout as Stochastic Regularization
Dropout is another common regularization technique. During training it randomly zeros out a fraction of activations, which prevents co adaptation of features and encourages redundancy. In PyTorch you add dropout as a module:
import torch.nn as nn
mlp = nn.Sequential(
nn.Linear(100, 256),
nn.ReLU(),
nn.Dropout(p=0.5),
nn.Linear(256, 10)
)
Dropout behaves differently in training and evaluation, which is controlled by model.train() and model.eval(). In evaluation mode, dropout is disabled, meaning all units are used.
If you forget to call model.eval() at validation or test time, dropout stays active and your metrics will be noisy and typically worse than they should be.
Early Stopping as a Practical Regularizer
Early stopping is a simple and very effective way to regularize: you stop training when validation performance stops improving. This limits how far the model can continue fitting noise in the training set.
A minimal pattern is to track the best validation loss and stop after a patience window:
best_val = float("inf")
patience = 5
bad_epochs = 0
for epoch in range(num_epochs):
model.train()
train_one_epoch()
model.eval()
val_loss = evaluate_val_loss()
if val_loss < best_val:
best_val = val_loss
bad_epochs = 0
best_state = {k: v.cpu().clone() for k, v in model.state_dict().items()}
else:
bad_epochs += 1
if bad_epochs >= patience:
breakEarly stopping is usually paired with checkpointing the best model weights, not the final epoch.
Data Augmentation as Regularization, The Idea
Another powerful form of regularization is to change the inputs during training in label preserving ways so the model sees many variations of the same underlying example. For images this might be random crops or flips, for text it might be token level noise, and for tabular data it might be noise injection or mixup style techniques. The details depend on the data type and are covered in the relevant data and vision chapters. The key concept here is that augmentation reduces overfitting by making memorization harder and by encouraging invariances you actually want.
How to Choose Regularization Strength
Regularization involves tradeoffs, and you typically tune it using validation performance. Weight decay values are often small for SGD and can be larger for AdamW, dropout probabilities are often between 0.1 and 0.5 depending on model size and dataset, and early stopping patience depends on how noisy validation metrics are.
A practical workflow is to introduce one regularizer at a time, confirm it reduces the gap between training and validation metrics, then tune its strength.
If your model is underfitting, regularization usually makes it worse. Underfitting signs include high training loss and low training accuracy that do not improve much with training.