KAHIBARO Bookshelves
Discord Login Register

4.4. Training Loop Structure

The Minimal Training Loop Mental Model

A PyTorch training loop is the repeated process of showing your model batches of data, measuring how wrong the predictions are, and adjusting the model parameters to reduce that error. The structure is always the same, even as models and datasets change. You initialize the model, loss function, and optimizer once, then you iterate over epochs, and inside each epoch you iterate over batches. Each batch does a forward pass to compute predictions, a loss computation, a backward pass to compute gradients, and an optimizer step to update weights.

Two modes matter in this structure. Training mode enables behaviors such as dropout and makes batch normalization update its running statistics, validation mode disables those training behaviors. You explicitly switch with model.train() and model.eval() and you also usually disable gradient tracking during validation so it runs faster and uses less memory.

The core order inside a training step is fixed: zero gradients, forward pass, compute loss, backward pass, optimizer step. If you change this order, you often get silent bugs.

One Epoch, Two Phases: Train Then Validate

A clean beginner friendly loop treats each epoch as having a training phase and an optional validation phase. During training you want gradients and parameter updates. During validation you want only evaluation of the current model, no gradient computation, no optimizer steps, and typically no randomness from training only layers.

In practice, the skeleton is: set model.train(), loop over training DataLoader, do update steps. Then set model.eval(), loop over validation DataLoader under torch.no_grad(), compute validation loss and metrics.

Always call model.eval() for validation and inference, and wrap validation code in with torch.no_grad():. Forgetting either can make validation noisy, slower, and can even change model state in batch normalization layers.

The Batch Update Step

Within the inner loop over batches, you do the same sequence every time. First, move the batch tensors to the same device as the model. Then reset gradients because PyTorch accumulates gradients by default. Then run the model on inputs to produce predictions. Then compute the loss by comparing predictions to targets. Then call loss.backward() to populate gradients in each parameter tensor. Finally, call optimizer.step() to update the parameters.

This step corresponds to minimizing an objective of the form
$$
\min_{\theta} \; \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(f_\theta(x_i), y_i),
$$
where $\theta$ are model parameters, $f_\theta$ is the model, and $\mathcal{L}$ is the loss function.

Gradients accumulate across backward calls. If you forget optimizer.zero_grad() (or set_to_none=True), your updates will effectively use summed gradients from multiple batches and training will behave incorrectly.

A Canonical PyTorch Skeleton

The following template is a solid default for beginner projects. It assumes you already have train_loader and optionally val_loader, plus a model, loss, and optimizer defined earlier in the chapter sequence.

python
import torch
def train_one_epoch(model, train_loader, optimizer, loss_fn, device):
    model.train()
    running_loss = 0.0
    for x, y in train_loader:
        x = x.to(device)
        y = y.to(device)
        optimizer.zero_grad(set_to_none=True)
        y_pred = model(x)
        loss = loss_fn(y_pred, y)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * x.size(0)
    epoch_loss = running_loss / len(train_loader.dataset)
    return epoch_loss
@torch.no_grad()
def validate(model, val_loader, loss_fn, device):
    model.eval()
    running_loss = 0.0
    for x, y in val_loader:
        x = x.to(device)
        y = y.to(device)
        y_pred = model(x)
        loss = loss_fn(y_pred, y)
        running_loss += loss.item() * x.size(0)
    epoch_loss = running_loss / len(val_loader.dataset)
    return epoch_loss
def fit(model, train_loader, val_loader, optimizer, loss_fn, device, epochs):
    model.to(device)
    for epoch in range(epochs):
        train_loss = train_one_epoch(model, train_loader, optimizer, loss_fn, device)
        if val_loader is not None:
            val_loss = validate(model, val_loader, loss_fn, device)
            print(f"epoch {epoch+1}/{epochs} train_loss={train_loss:.4f} val_loss={val_loss:.4f}")
        else:
            print(f"epoch {epoch+1}/{epochs} train_loss={train_loss:.4f}")

This structure makes two important ideas explicit. First, the model is moved to the device once in fit, not repeatedly. Second, training and validation are separated so you do not accidentally update weights during validation.

Where People Commonly Break the Structure

Beginners often compute loss correctly but forget that only optimizer.step() changes the model. Without it, the model never learns. Another common mistake is calling loss.backward() during validation or leaving model.train() on, which makes validation results inconsistent.

You should also be careful about what you average. loss.item() is a per batch scalar that may already be a mean over the batch, depending on the loss function reduction. Multiplying by batch_size and dividing by the dataset size is a reliable way to get an epoch average that is not biased by the final smaller batch.

Do not call optimizer.step() in validation, and do not compute validation loss with gradients enabled. Validation should never modify model parameters or accumulate gradients.

A Quick Checklist for a Healthy Loop

At the start of each training epoch you set training mode. For every batch you move data to the device, clear gradients, compute predictions, compute loss, backpropagate, then step the optimizer. After training, you switch to eval mode for validation, disable gradients, and only compute losses and metrics. If you can recite that sequence and your code matches it, you have the correct training loop structure that scales to nearly every model you will build later in the course.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics