6.1. Forward Pass and Backward Pass
Table of Contents
What the forward pass is in practice
In a PyTorch training step, the forward pass is the part where you feed a batch of inputs into the model to produce predictions, then compute a scalar loss that measures how wrong those predictions are. Concretely, you call the model like a function, such as y_pred = model(x), and then compute something like loss = loss_fn(y_pred, y). The forward pass is also where PyTorch records the operations needed for automatic differentiation, as long as gradient tracking is enabled.
A useful mental model is that the forward pass builds a computation history that connects the loss back to the model parameters. In supervised learning you typically want a single scalar loss, because that is what you will differentiate with respect to the parameters.
What the backward pass is in practice
The backward pass is where you ask PyTorch to compute gradients of the loss with respect to all trainable parameters. You do this by calling loss.backward(). Under the hood, PyTorch applies the chain rule to propagate derivatives from the loss back through the operations performed during the forward pass, accumulating a .grad tensor for each parameter.
If $L$ is the loss and $\theta$ represents a parameter tensor, the backward pass computes $\frac{\partial L}{\partial \theta}$ and stores it in theta.grad. Those gradients are then consumed by the optimizer step, which is covered elsewhere.
Rule: loss.backward() computes gradients for the current computation graph, and stores them in each parameter’s .grad field. Gradients accumulate by default if you call backward() multiple times without clearing .grad.
A minimal training step with forward and backward
A typical single iteration looks like this:
model.train()
x, y = batch
x = x.to(device)
y = y.to(device)
y_pred = model(x) # forward pass
loss = loss_fn(y_pred, y) # still part of forward
loss.backward() # backward pass
optimizer.step() # parameter update, covered in the optimizer chapter
Everything up to and including loss = ... is the forward computation. The gradients are not computed until loss.backward() is called. After backward, each parameter in model.parameters() that has requires_grad=True will typically have a non None .grad tensor.
How autograd knows what to differentiate
PyTorch autograd tracks operations on tensors that have requires_grad=True, and it tracks any tensor that is produced from them through differentiable operations. Model parameters created by nn.Module layers usually have requires_grad=True automatically. Your input data x usually does not need gradients, so it typically has requires_grad=False, which is fine because gradients are needed for parameters, not for inputs, in standard training.
During the forward pass, the intermediate tensors store references to how they were computed. When you call loss.backward(), autograd traverses those references backward to compute gradients.
Rule: If you accidentally disable gradient tracking during training, for example by wrapping the forward pass in torch.no_grad(), then loss.backward() will fail or produce no gradients because the graph was not recorded.
Scalar losses, gradient outputs, and common shape expectations
Most beginner training loops use a scalar loss such as mean squared error or cross entropy reduction, so loss has shape [], meaning a zero dimensional scalar tensor. In that common case, loss.backward() needs no arguments.
If your loss is not a scalar, you must provide the gradient of some scalar objective with respect to that tensor, using loss.backward(gradient=...). In practice, beginners should keep the loss reduced to a scalar by using the default reduction or explicitly averaging or summing.
Rule: Prefer a scalar loss. If loss is not scalar, loss.backward() requires an explicit gradient argument, and mistakes here often lead to confusing errors.
Retaining graphs and multiple backward passes
By default, after loss.backward(), PyTorch frees the saved intermediate buffers for that computation graph to save memory. If you need to call backward more than once on the same graph, you must set retain_graph=True on the first backward call. This is uncommon in basic training, but can appear in advanced setups like certain multi loss objectives or higher order derivatives.
A typical beginner error is calling loss.backward() twice on the same loss without retaining the graph, which triggers an error about trying to backward through the graph a second time.
Rule: You normally call backward() once per forward pass. If you truly need multiple backward calls on the same forward, you must use retain_graph=True, and you should expect higher memory use.
Gradient accumulation is a feature, not a bug
Each call to loss.backward() adds gradients into existing .grad buffers. This is why gradient zeroing is necessary and is handled in the separate chapter on zeroing and accumulation. Here, the key point is conceptual: backward does not overwrite gradients by default, it accumulates them.
This accumulation can be used intentionally, for example to simulate a larger batch by running several forward and backward passes before applying an optimizer step, but if you do not plan for it, it will silently change training behavior.
Training mode, inference mode, and where they matter
Forward and backward passes occur during training, but some layers behave differently depending on whether the model is in training mode or evaluation mode. You set this with model.train() and model.eval(). The backward pass itself only cares that a graph exists, but the forward pass can change depending on mode, which affects the loss and thus the gradients.
For evaluation or inference, you typically use model.eval() and disable gradient tracking to save memory and compute, but you should not do that during training forward passes that need gradients.
Rule: During training, do the forward pass with gradient tracking enabled, and use model.train() so train time layer behavior is correct. During evaluation, use model.eval() and usually torch.no_grad().
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!