KAHIBARO Bookshelves
Discord Login Register

9.6. Fine-Tuning and Freezing Layers

What Freezing and Fine-Tuning Mean

When you use a pretrained vision model, most of the useful visual features are already stored in its weights. Freezing means you prevent some parameters from being updated during training. Fine-tuning means you allow parameters to update so the model adapts to your dataset. In practice you often replace the model head for your task, then decide how much of the backbone to freeze or fine-tune.

A common pattern is to freeze the entire backbone at first and train only a newly added classifier head. If results plateau, you progressively unfreeze later layers and fine-tune them with a smaller learning rate.

Replacing the Classification Head

Pretrained models are typically trained on ImageNet with 1000 classes, so the last layer does not match your number of classes. You usually keep the backbone and swap the final classifier layer.

In torchvision, different architectures expose the classifier differently. For example, ResNet uses model.fc, many MobileNet and EfficientNet variants use model.classifier, and VGG uses model.classifier as a sequential block. The key idea is always the same, you create a new nn.Linear with output size equal to your classes.

Important rule: you must replace the head before constructing the optimizer, otherwise the optimizer might not include the new parameters.

Freezing Parameters Correctly

In PyTorch, a parameter is updated by gradient descent only if two things happen, it has requires_grad=True, and it is included in the optimizer parameter list. Freezing is usually done by setting requires_grad=False for selected parameters.

A typical workflow is to freeze everything, then unfreeze just the head.

python
import torch
import torch.nn as nn
from torchvision import models
num_classes = 5
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
for p in model.parameters():
    p.requires_grad = False
in_features = model.fc.in_features
model.fc = nn.Linear(in_features, num_classes)
for p in model.fc.parameters():
    p.requires_grad = True

At this point, only the head will produce gradients.

Important rule: freezing with requires_grad=False stops gradient computation for those parameters, but you still must ensure your optimizer only receives the parameters you intend to train.

Building the Optimizer for Only Trainable Parameters

To train only the head, pass only trainable parameters to the optimizer. A robust pattern is filtering by requires_grad.

python
optimizer = torch.optim.Adam(
    (p for p in model.parameters() if p.requires_grad),
    lr=1e-3
)

This avoids accidentally training frozen layers and automatically includes new layers you added.

Unfreezing in Stages

If your dataset is moderately sized and different from ImageNet, training only the head may not be enough. A common next step is to unfreeze the last block or the last few layers of the backbone and fine-tune them. With ResNet, later blocks are often layer4, then layer3, and so on.

python
for p in model.layer4.parameters():
    p.requires_grad = True

After changing which parameters require gradients, you should recreate the optimizer so it includes the newly trainable parameters.

Important rule: when you unfreeze additional layers, recreate the optimizer, otherwise those parameters will not be updated because the optimizer does not know about them.

Using Different Learning Rates for Head and Backbone

Newly initialized layers like your head usually need a larger learning rate than pretrained layers, which you want to change more gently. You can set parameter groups in the optimizer.

python
head_params = list(model.fc.parameters())
backbone_params = [p for n, p in model.named_parameters()
                   if not n.startswith("fc.") and p.requires_grad]
optimizer = torch.optim.Adam([
    {"params": backbone_params, "lr": 1e-5},
    {"params": head_params, "lr": 1e-3},
])

This is one of the most practical tricks for stable fine-tuning.

Important rule: use a smaller learning rate for pretrained layers than for the newly initialized head, often 10 times to 100 times smaller.

BatchNorm and the Difference Between Freezing Weights and Freezing Behavior

Layers like BatchNorm have parameters and also internal running statistics that update during training when the model is in train() mode. Even if you freeze BatchNorm parameters with requires_grad=False, the running mean and variance can still change during training, which can hurt when your dataset is small.

A common approach is to keep the model in training mode overall but set BatchNorm layers to evaluation mode so their running statistics do not update.

python
def set_bn_eval(m):
    if isinstance(m, nn.BatchNorm2d):
        m.eval()
model.apply(set_bn_eval)

This is not always required, but it is a frequent fix when fine-tuning becomes unstable on small datasets.

Important rule: requires_grad=False freezes gradients, but it does not automatically stop BatchNorm running statistics updates. Control this with train() and eval() behavior.

When to Freeze vs Fine-Tune

Freezing most of the network is often best when your dataset is small, similar to the pretrained domain, or you want fast training and lower risk of overfitting. Fine-tuning more layers is often best when your dataset is larger, significantly different from ImageNet, or you need extra accuracy and can afford more compute.

A practical heuristic is to start with head only, then progressively unfreeze from the end of the backbone, watching validation performance.

A Minimal Fine-Tuning Checklist

Before you start training, confirm that the head output matches the number of classes, that only intended parameters have requires_grad=True, that the optimizer was created after freezing and head replacement, and that you are using a smaller learning rate for pretrained layers if you fine-tune them. During training, if validation accuracy drops sharply when you unfreeze, reduce the backbone learning rate and consider freezing BatchNorm behavior.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics