9.6. Fine-Tuning and Freezing Layers
Table of Contents
What Freezing and Fine-Tuning Mean
When you use a pretrained vision model, most of the useful visual features are already stored in its weights. Freezing means you prevent some parameters from being updated during training. Fine-tuning means you allow parameters to update so the model adapts to your dataset. In practice you often replace the model head for your task, then decide how much of the backbone to freeze or fine-tune.
A common pattern is to freeze the entire backbone at first and train only a newly added classifier head. If results plateau, you progressively unfreeze later layers and fine-tune them with a smaller learning rate.
Replacing the Classification Head
Pretrained models are typically trained on ImageNet with 1000 classes, so the last layer does not match your number of classes. You usually keep the backbone and swap the final classifier layer.
In torchvision, different architectures expose the classifier differently. For example, ResNet uses model.fc, many MobileNet and EfficientNet variants use model.classifier, and VGG uses model.classifier as a sequential block. The key idea is always the same, you create a new nn.Linear with output size equal to your classes.
Important rule: you must replace the head before constructing the optimizer, otherwise the optimizer might not include the new parameters.
Freezing Parameters Correctly
In PyTorch, a parameter is updated by gradient descent only if two things happen, it has requires_grad=True, and it is included in the optimizer parameter list. Freezing is usually done by setting requires_grad=False for selected parameters.
A typical workflow is to freeze everything, then unfreeze just the head.
import torch
import torch.nn as nn
from torchvision import models
num_classes = 5
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
for p in model.parameters():
p.requires_grad = False
in_features = model.fc.in_features
model.fc = nn.Linear(in_features, num_classes)
for p in model.fc.parameters():
p.requires_grad = TrueAt this point, only the head will produce gradients.
Important rule: freezing with requires_grad=False stops gradient computation for those parameters, but you still must ensure your optimizer only receives the parameters you intend to train.
Building the Optimizer for Only Trainable Parameters
To train only the head, pass only trainable parameters to the optimizer. A robust pattern is filtering by requires_grad.
optimizer = torch.optim.Adam(
(p for p in model.parameters() if p.requires_grad),
lr=1e-3
)This avoids accidentally training frozen layers and automatically includes new layers you added.
Unfreezing in Stages
If your dataset is moderately sized and different from ImageNet, training only the head may not be enough. A common next step is to unfreeze the last block or the last few layers of the backbone and fine-tune them. With ResNet, later blocks are often layer4, then layer3, and so on.
for p in model.layer4.parameters():
p.requires_grad = TrueAfter changing which parameters require gradients, you should recreate the optimizer so it includes the newly trainable parameters.
Important rule: when you unfreeze additional layers, recreate the optimizer, otherwise those parameters will not be updated because the optimizer does not know about them.
Using Different Learning Rates for Head and Backbone
Newly initialized layers like your head usually need a larger learning rate than pretrained layers, which you want to change more gently. You can set parameter groups in the optimizer.
head_params = list(model.fc.parameters())
backbone_params = [p for n, p in model.named_parameters()
if not n.startswith("fc.") and p.requires_grad]
optimizer = torch.optim.Adam([
{"params": backbone_params, "lr": 1e-5},
{"params": head_params, "lr": 1e-3},
])This is one of the most practical tricks for stable fine-tuning.
Important rule: use a smaller learning rate for pretrained layers than for the newly initialized head, often 10 times to 100 times smaller.
BatchNorm and the Difference Between Freezing Weights and Freezing Behavior
Layers like BatchNorm have parameters and also internal running statistics that update during training when the model is in train() mode. Even if you freeze BatchNorm parameters with requires_grad=False, the running mean and variance can still change during training, which can hurt when your dataset is small.
A common approach is to keep the model in training mode overall but set BatchNorm layers to evaluation mode so their running statistics do not update.
def set_bn_eval(m):
if isinstance(m, nn.BatchNorm2d):
m.eval()
model.apply(set_bn_eval)This is not always required, but it is a frequent fix when fine-tuning becomes unstable on small datasets.
Important rule: requires_grad=False freezes gradients, but it does not automatically stop BatchNorm running statistics updates. Control this with train() and eval() behavior.
When to Freeze vs Fine-Tune
Freezing most of the network is often best when your dataset is small, similar to the pretrained domain, or you want fast training and lower risk of overfitting. Fine-tuning more layers is often best when your dataset is larger, significantly different from ImageNet, or you need extra accuracy and can afford more compute.
A practical heuristic is to start with head only, then progressively unfreeze from the end of the backbone, watching validation performance.
A Minimal Fine-Tuning Checklist
Before you start training, confirm that the head output matches the number of classes, that only intended parameters have requires_grad=True, that the optimizer was created after freezing and head replacement, and that you are using a smaller learning rate for pretrained layers if you fine-tune them. During training, if validation accuracy drops sharply when you unfreeze, reduce the backbone learning rate and consider freezing BatchNorm behavior.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!