KAHIBARO Bookshelves
Discord Login Register

13.1. Saving Model Weights and Full Checkpoints

What to Save and Why

In PyTorch you typically save either model weights only or a full training checkpoint. Saving weights only means saving the parameters of your neural network, usually with model.state_dict(). This is the most common choice for deployment and inference because it is compact and portable across training setups. A full checkpoint saves enough information to resume training later as if nothing happened. That usually includes the model weights, optimizer state, learning rate scheduler state, the current epoch or step, and any metadata you need to reproduce the run.

Rule of thumb: save weights for inference, save full checkpoints for resuming training.

Saving Model Weights with `state_dict`

A model’s state_dict is a Python dictionary that maps parameter and buffer names to tensors. You save it with torch.save and later load it into an identically constructed model.

python
import torch
# during or after training
torch.save(model.state_dict(), "model_weights.pt")

When loading, you must first create the model object with the same architecture, then load the weights.

python
model = MyModel(**same_constructor_args)
state = torch.load("model_weights.pt", map_location="cpu")
model.load_state_dict(state)
model.eval()

Calling model.eval() matters for inference because layers like dropout and batch normalization behave differently in training mode versus evaluation mode.

Always reconstruct the model architecture before calling load_state_dict. The saved state_dict does not contain the Python code that defines your model.

Saving Full Checkpoints for Resuming Training

A full checkpoint is usually a dictionary you define yourself. At minimum it includes the model and optimizer states. If you use a learning rate scheduler, include that too. It is also common to store the epoch or global step and the best validation metric so far.

python
checkpoint = {
    "epoch": epoch,
    "model_state": model.state_dict(),
    "optimizer_state": optimizer.state_dict(),
    "scheduler_state": scheduler.state_dict() if scheduler is not None else None,
    "best_val_loss": best_val_loss,
}
torch.save(checkpoint, "checkpoint.pt")

To resume, you load the dictionary, restore states, and continue from the saved epoch.

python
ckpt = torch.load("checkpoint.pt", map_location=device)
model.load_state_dict(ckpt["model_state"])
optimizer.load_state_dict(ckpt["optimizer_state"])
if scheduler is not None and ckpt["scheduler_state"] is not None:
    scheduler.load_state_dict(ckpt["scheduler_state"])
start_epoch = ckpt["epoch"] + 1
best_val_loss = ckpt.get("best_val_loss", float("inf"))

If you want to resume training faithfully, you must save and load the optimizer state. Without it, momentum based optimizers like Adam will behave as if training just started, even if the weights look correct.

Device and `map_location` Details

Checkpoints can be saved on one device and loaded on another. Use map_location to control where tensors land when loading. A common safe pattern is to load on CPU first, then move the model to the target device.

python
ckpt = torch.load("checkpoint.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state"])
model.to(device)

This reduces the chance of GPU memory spikes during loading.

Naming Conventions and What to Keep

It helps to maintain two kinds of files: a latest checkpoint that you overwrite each epoch, and a best checkpoint that you only update when validation improves. The latest file is for recovery from interruptions, the best file is for evaluation and deployment.

Do not rely on only one checkpoint file. Keep at least a latest and a best to protect against crashes and bad epochs.

Common Pitfalls

One frequent issue is shape mismatches when loading weights, which usually means the model definition changed. Another is forgetting to save non model state needed to resume training, such as the scheduler. Also note that some layers register non parameter buffers, for example batch normalization running statistics, and these are included in state_dict, which is one reason state_dict is preferred over saving individual parameter tensors.

When something does not load, inspect the keys.

python
state = torch.load("model_weights.pt", map_location="cpu")
print(list(state.keys())[:10])

Recommended Minimal Templates

For inference, the minimal template is saving model.state_dict() and separately recording the model constructor arguments or configuration in a JSON or Python dict. For training resumption, the minimal template is a dict with model_state, optimizer_state, and an epoch or step counter, plus scheduler state if you use one.

A checkpoint is only as useful as the metadata you store with it. If you cannot reconstruct the model configuration, the weights are not reusable.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics