6.3. Device Placement and Mixed Precision Basics
Table of Contents
Moving Tensors and Models to the Right Device
In PyTorch, every tensor lives on a device, most commonly the CPU or a GPU. A model is just a collection of parameters, and each parameter is a tensor that also lives on a device. The first practical rule for training loops is that the model parameters and every tensor used in the forward pass must be on the same device, otherwise PyTorch will raise a runtime error.
A typical pattern is to pick a device once, move the model to it, then ensure each batch is moved to that same device right before the forward pass. In code, this usually means defining device = torch.device("cuda") when CUDA is available, otherwise falling back to torch.device("cpu"), then calling model.to(device). For each batch, you call x = x.to(device) and y = y.to(device).
All tensors that interact in an operation must be on the same device. If your model is on CUDA, your inputs and targets must be moved to CUDA before the forward pass.
When moving batches, you will often see x = x.to(device, non_blocking=True) when using a DataLoader with pinned memory. That combination can reduce input transfer overhead when training on GPU, but it is an optimization, not a requirement.
Checking and Debugging Device Issues
Device mistakes are among the most common beginner issues because they can hide until the first real batch hits a layer. It helps to verify early. You can check a tensor’s device via x.device and a model parameter device via next(model.parameters()).device. If those do not match, fix that first.
Another frequent source of confusion is creating new tensors inside the training step. For example, calling torch.zeros(...) without specifying a device creates a CPU tensor by default. If you need a new tensor during the forward pass or loss computation, create it on the same device as existing tensors, often by using torch.zeros_like(existing_tensor) or torch.zeros(..., device=x.device).
New tensors you create inside the training loop default to CPU unless you explicitly place them on the correct device. Prefer *_like helpers to automatically match device and dtype.
What Mixed Precision Is and Why It Helps
Mixed precision training uses a combination of floating point formats, typically FP16 or BF16 for many computations, while keeping some values in FP32 for numerical stability. The main benefits on modern GPUs are faster throughput and lower memory usage, which can allow larger batch sizes or bigger models.
Two concepts matter in practice. The first is autocasting, which tells PyTorch to run certain operations in lower precision when it is safe. The second is gradient scaling, which helps avoid underflow in FP16 gradients by multiplying the loss by a scale factor during backpropagation, then unscaling before the optimizer step.
BF16 is generally more numerically stable than FP16 because it has a larger exponent range. On supported hardware, BF16 often needs no gradient scaling. FP16 usually benefits from gradient scaling.
Using `torch.cuda.amp` in a Training Loop
The standard PyTorch approach for mixed precision on CUDA uses torch.autocast and torch.cuda.amp.GradScaler. You keep model parameters in FP32, and you wrap the forward and loss computation in an autocast context. Backward uses scaled loss, then you step the optimizer through the scaler.
A minimal training step looks like this:
scaler = torch.cuda.amp.GradScaler(enabled=(device.type == "cuda"))
for x, y in train_loader:
x = x.to(device, non_blocking=True)
y = y.to(device, non_blocking=True)
optimizer.zero_grad(set_to_none=True)
with torch.autocast(device_type="cuda", dtype=torch.float16, enabled=(device.type == "cuda")):
preds = model(x)
loss = loss_fn(preds, y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()This pattern works because autocast affects forward computations, while GradScaler manages the backward and step safely for FP16. If you are not on CUDA, you typically disable autocast and scaling so the same code path still runs.
When using FP16 mixed precision, do not call optimizer.step() directly. Use scaler.step(optimizer) and scaler.update() so that gradients are properly unscaled and invalid steps can be skipped.
Mixed Precision on CPU and Apple Silicon
Mixed precision is primarily beneficial on GPUs. Autocast also exists for CPU, but the speedups are usually smaller and depend heavily on hardware and workload. On Apple Silicon, PyTorch uses the MPS backend for GPU acceleration, and mixed precision support and behavior can differ from CUDA. For absolute beginners, the simplest, most reliable approach is to use mixed precision only when training on CUDA, and to keep CPU and MPS training in full precision unless you have a reason and have verified stability.
Common Stability Problems and How to React
If your loss becomes NaN or training diverges after enabling mixed precision, first confirm that your non mixed precision run is stable. Then try switching FP16 to BF16 if your GPU supports it, because BF16 is often more forgiving. Another safe fallback is to disable mixed precision entirely for that experiment. Mixed precision is an optimization, not a requirement.
Also be careful with operations known to be numerically sensitive, such as certain reductions, exponentials, or softmax in custom code. Autocast handles many cases well, but if you write custom math, you may need to compute sensitive parts in FP32.
Mixed precision should not change your model’s correctness goals. If enabling it makes training unstable, switch to BF16 if available or disable mixed precision and continue with FP32.
Practical Rules of Thumb for Beginners
Pick one device variable and use it everywhere, move the model once, and move each batch right before the forward pass. Avoid creating stray CPU tensors inside the training step. Enable CUDA mixed precision only after your baseline training loop works, then adopt the standard autocast plus GradScaler pattern. Keep the code structured so you can flip mixed precision on or off with a single flag, which makes debugging much easier.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!