KAHIBARO Bookshelves
Discord Login Register

3.5. Regularization Basics

What Regularization Is and Why You Need It

Regularization is anything you add to training that makes a model less likely to memorize the training set and more likely to generalize to new data. In practice, regularization usually increases training loss a bit while reducing validation loss and improving real world performance. You will notice regularization matters most when the model has enough capacity to overfit, for example many parameters relative to the dataset size, high input dimensionality, noisy labels, or too many training epochs.

A useful mental model is that you are optimizing not just the data fitting term, but a slightly modified objective that discourages overly complex solutions. For many common methods, this looks like

$$
\mathcal{L}_{total}(\theta) = \mathcal{L}_{data}(\theta) + \lambda \, \Omega(\theta)
$$

where $\theta$ are the parameters, $\Omega(\theta)$ is a penalty that grows when the model gets too complex, and $\lambda$ controls how strong the penalty is.

Regularization is not a substitute for a validation set. You still need validation to choose regularization strength and to detect overfitting.

L2 Regularization, Weight Decay in Optimizers

The most common regularization for neural networks is L2 regularization on weights, which penalizes large weights via $\Omega(\theta) = \lVert \theta \rVert_2^2$. The key practical concept in PyTorch is that you typically apply it through the optimizer as weight decay.

With L2 regularization, the objective becomes:

$$
\mathcal{L}_{total} = \mathcal{L}_{data} + \lambda \sum_i \theta_i^2
$$

In PyTorch, many optimizers accept a weight_decay argument. For example, with SGD:

python
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=1e-4)

For Adam type optimizers, it is usually better to use AdamW, which implements decoupled weight decay:

python
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)

Decoupled weight decay tends to behave more predictably because it applies the decay separately from the adaptive gradient update.

Do not assume weight_decay is always identical to classic L2 regularization for every optimizer. If you use Adam, prefer AdamW when you want weight decay regularization.

What Not to Decay, Biases and Normalization Parameters

In many models, you should avoid applying weight decay to bias terms and to parameters of normalization layers. The reason is that these parameters do not control complexity in the same way as weight matrices, and decaying them can sometimes hurt optimization or accuracy.

A typical approach is to create parameter groups, one with decay and one without:

python
decay, no_decay = [], []
for name, param in model.named_parameters():
    if not param.requires_grad:
        continue
    if name.endswith(".bias") or "bn" in name.lower() or "norm" in name.lower():
        no_decay.append(param)
    else:
        decay.append(param)
optimizer = torch.optim.AdamW(
    [{"params": decay, "weight_decay": 1e-2},
     {"params": no_decay, "weight_decay": 0.0}],
    lr=3e-4
)

This pattern becomes more important as you move to modern architectures that use many normalization layers.

If you apply weight decay to everything by default, you may unintentionally regularize biases and normalization parameters, which can reduce performance.

Dropout as Stochastic Regularization

Dropout is another common regularization technique. During training it randomly zeros out a fraction of activations, which prevents co adaptation of features and encourages redundancy. In PyTorch you add dropout as a module:

python
import torch.nn as nn
mlp = nn.Sequential(
    nn.Linear(100, 256),
    nn.ReLU(),
    nn.Dropout(p=0.5),
    nn.Linear(256, 10)
)

Dropout behaves differently in training and evaluation, which is controlled by model.train() and model.eval(). In evaluation mode, dropout is disabled, meaning all units are used.

If you forget to call model.eval() at validation or test time, dropout stays active and your metrics will be noisy and typically worse than they should be.

Early Stopping as a Practical Regularizer

Early stopping is a simple and very effective way to regularize: you stop training when validation performance stops improving. This limits how far the model can continue fitting noise in the training set.

A minimal pattern is to track the best validation loss and stop after a patience window:

python
best_val = float("inf")
patience = 5
bad_epochs = 0
for epoch in range(num_epochs):
    model.train()
    train_one_epoch()
    model.eval()
    val_loss = evaluate_val_loss()
    if val_loss < best_val:
        best_val = val_loss
        bad_epochs = 0
        best_state = {k: v.cpu().clone() for k, v in model.state_dict().items()}
    else:
        bad_epochs += 1
        if bad_epochs >= patience:
            break

Early stopping is usually paired with checkpointing the best model weights, not the final epoch.

Data Augmentation as Regularization, The Idea

Another powerful form of regularization is to change the inputs during training in label preserving ways so the model sees many variations of the same underlying example. For images this might be random crops or flips, for text it might be token level noise, and for tabular data it might be noise injection or mixup style techniques. The details depend on the data type and are covered in the relevant data and vision chapters. The key concept here is that augmentation reduces overfitting by making memorization harder and by encouraging invariances you actually want.

How to Choose Regularization Strength

Regularization involves tradeoffs, and you typically tune it using validation performance. Weight decay values are often small for SGD and can be larger for AdamW, dropout probabilities are often between 0.1 and 0.5 depending on model size and dataset, and early stopping patience depends on how noisy validation metrics are.

A practical workflow is to introduce one regularizer at a time, confirm it reduces the gap between training and validation metrics, then tune its strength.

If your model is underfitting, regularization usually makes it worse. Underfitting signs include high training loss and low training accuracy that do not improve much with training.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics