KAHIBARO Bookshelves
Discord Login Register

14.6. Practical Guidelines for Faster Training

Measure First, Then Change One Thing

Faster training starts with knowing what is slow. In practice, the bottleneck is usually either the input pipeline, the model compute, or synchronization between CPU and GPU. The most reliable workflow is to change one variable at a time and confirm the effect with timing, because many speed tips interact and some can even slow you down depending on your hardware and batch size.

A simple timing approach is to measure wall clock time per iteration and per epoch after a short warmup, because the first few iterations include one time setup costs like CUDA kernel compilation and caching. When measuring GPU work, remember that CUDA operations are asynchronous, so you only get meaningful timings if you synchronize before stopping the timer. If you forget this, you can mistakenly conclude that a change sped things up when you only changed when the program waits.

Rule: For accurate GPU timing, insert torch.cuda.synchronize() before starting and before stopping the timer. Otherwise timings can be misleading because CUDA launches are asynchronous.

Use the Largest Batch Size That Fits

Batch size is often the biggest lever for throughput because it increases parallel work per step and reduces overhead per sample. The limit is usually GPU memory. A practical approach is to increase batch size until you get an out of memory error, then back off slightly. When you increase batch size, you may need to adjust the learning rate schedule. The details of learning rate tuning belong elsewhere, but the key speed point is that larger batches often reduce total steps per epoch, which reduces overhead from Python and logging.

If memory is the only thing preventing a larger batch, use gradient accumulation to simulate a larger effective batch while keeping micro batches small enough to fit. This increases compute time per optimizer step, but can improve overall utilization and reduce time spent on optimizer overhead and dataloader overhead per sample.

Prefer Mixed Precision on GPUs That Support It

On modern NVIDIA GPUs, mixed precision training can significantly increase throughput and reduce memory use. In PyTorch, this usually means using torch.cuda.amp.autocast for the forward pass and a GradScaler to safely scale gradients. The main speed win comes from using Tensor Cores and reduced memory bandwidth pressure. Mixed precision is not always faster on every model, but it often is for convolutional networks and transformers.

Be careful when comparing runs, because mixed precision can change numerics slightly. If you are seeing instability, verify loss scaling and watch for NaNs, then consider disabling mixed precision temporarily to confirm whether it is the cause.

Rule: Only judge mixed precision speed after a warmup and with correct synchronization timing, because kernel selection and caching can skew first iteration measurements.

Reduce Python Overhead in the Training Step

Python level overhead becomes visible when your model is small or your batch size is small. A few practical habits help.

Keep the per iteration work minimal. Avoid building new tensors in Python loops when you can use vectorized tensor operations. Avoid frequent .item() calls, because they often force synchronization between CPU and GPU. Log less frequently, for example every N steps, and prefer aggregating metrics on GPU and transferring to CPU occasionally.

Set the model to the right mode. Use model.train() during training and model.eval() during evaluation, because layers like dropout and batch norm behave differently and can affect performance and correctness.

If you are on PyTorch 2.x, torch.compile can speed up many models by reducing Python overhead and optimizing graphs, although the payoff depends on the model and the backend. It also adds compile time overhead, so it tends to help more for long training runs than for quick experiments.

Keep the GPU Busy by Feeding Data Efficiently

Even a very fast GPU will sit idle if it waits for batches. When training seems slow and GPU utilization is low, the data pipeline is a common culprit. Use multiple DataLoader workers so preprocessing runs in parallel, and enable pinned memory so host to device transfers are faster. When moving batches to GPU, using non blocking transfers can overlap copies with compute when the rest of the pipeline is configured correctly.

If preprocessing is expensive, consider caching preprocessed data when possible, or moving lightweight transforms onto the GPU. If the dataset is stored on a slow drive, storage can become the bottleneck, so placing data on a fast SSD can make a bigger difference than code tweaks.

Rule: If GPU utilization is low, do not optimize the model first. Fix the input pipeline and host to device transfers so the GPU is consistently fed with data.

Choose Efficient Model and Tensor Operations

Small implementation choices can change performance. Prefer operations that are fused or highly optimized. For example, use built in PyTorch layers and losses rather than manual implementations when possible, because they are typically optimized and may use fused kernels.

Pay attention to tensor memory layout. Many operations are fastest when tensors are contiguous. Excessive transposes and slices can create non contiguous tensors that force implicit copies later. If you see unexpected slowdowns after indexing or permuting, check whether calling .contiguous() at the right point improves performance, but only do this if you have measured that contiguity is the issue, because extra copies can also hurt.

Also minimize unnecessary dtype conversions. Keep data in the dtype you intend to compute with, and avoid repeatedly converting inside the loop.

Avoid Unnecessary Synchronization and Transfers

A common hidden slowdown is moving data between CPU and GPU too often. Keep tensors on the GPU during the training step, and only move results to CPU when needed for logging or saving. Avoid calling .cpu() or .numpy() inside the loop unless necessary, and avoid frequent printing, which can dominate runtime.

When computing metrics, prefer computing them on the GPU and reducing to a scalar occasionally. If you must call .item(), do it sparingly.

Tune Validation and Checkpoint Frequency

Validation and checkpointing are essential, but they can be expensive. If you validate too often, training can appear slow even if the training step is fast. A practical approach is to validate every fixed number of steps or every epoch early on, then less frequently once the run is stable, depending on how quickly the model changes.

Checkpointing can also stall training, especially if you save large optimizer states. Save the minimal state you need at the frequency you need, and consider saving asynchronously in a separate process if you have very strict throughput requirements.

Use Early Stopping for Time to Result

If your goal is to reach a target validation score quickly, the fastest training is often training less. Early stopping based on validation metrics can save substantial time. This is not a compute optimization, but it is a workflow optimization that reduces wasted epochs once the model stops improving.

A Practical Speed Checklist

Start by confirming that the GPU is actually the bottleneck, then increase batch size until memory is full, enable mixed precision if appropriate, and make sure the DataLoader keeps up. Reduce Python overhead by logging less and avoiding synchronizing calls. Only then consider heavier tools like compilation, advanced kernel optimizations, or architectural changes. The consistent theme is to measure, change one thing, and verify the improvement with correct timing.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Orientation and Learning Setup
    • ☰ 1.1. Course Goals and How to Use This Outline
      • ☰ 1.2. What You Need to Know Before Starting
        • ☰ 1.3. Installing Python and PyTorch
          • ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
            • ☰ 1.5. Using Jupyter, VS Code, and Scripts
              • ☰ 1.6. Reproducibility, Seeds, and Determinism
              • ☰ 2. Python and PyTorch Fundamentals
                • ☰ 2.1. Tensors and Basic Operations
                  • ☰ 2.2. Tensor Shapes, Dtypes, and Devices
                    • ☰ 2.3. Indexing, Slicing, and Broadcasting
                      • ☰ 2.4. Autograd and Computational Graphs
                        • ☰ 2.5. Common Tensor Pitfalls and Debugging
                          • ☰ 2.6. Saving and Loading Tensors
                          • ☰ 3. Neural Network Building Blocks
                            • ☰ 3.1. Linear Layers and Affine Transforms
                              • ☰ 3.2. Activation Functions
                                • ☰ 3.3. Loss Functions
                                  • ☰ 3.4. Optimizers and Learning Rates
                                    • ☰ 3.5. Regularization Basics
                                      • ☰ 3.6. Weight Initialization Concepts
                                      • ☰ 4. First Models End to End
                                        • ☰ 4.1. A Simple Regression Model
                                          • ☰ 4.2. A Simple Classification Model
                                            • ☰ 4.3. Data Splitting and Validation Sets
                                              • ☰ 4.4. Training Loop Structure
                                                • ☰ 4.5. Tracking Loss and Metrics
                                                  • ☰ 4.6. Basic Overfitting and Underfitting Checks
                                                  • ☰ 5. Data Handling with PyTorch
                                                    • ☰ 5.1. Datasets and DataLoaders
                                                      • ☰ 5.2. Transforms and Preprocessing
                                                        • ☰ 5.3. Collate Functions and Batching
                                                          • ☰ 5.4. Shuffling, Sampling, and Class Imbalance
                                                            • ☰ 5.5. Working with Tabular, Text, and Image Data
                                                              • ☰ 5.6. Data Pipeline Performance Basics
                                                              • ☰ 6. Training Loops in Practice
                                                                • ☰ 6.1. Forward Pass and Backward Pass
                                                                  • ☰ 6.2. Gradient Zeroing and Accumulation
                                                                    • ☰ 6.3. Device Placement and Mixed Precision Basics
                                                                      • ☰ 6.4. Learning Rate Schedules
                                                                        • ☰ 6.5. Gradient Clipping
                                                                          • ☰ 6.6. Logging with TensorBoard
                                                                          • ☰ 7. Debugging and Model Troubleshooting
                                                                            • ☰ 7.1. Verifying Shapes and NaNs
                                                                              • ☰ 7.2. Checking Gradients and Vanishing Gradients
                                                                                • ☰ 7.3. Sanity Checks and Overfit a Small Batch
                                                                                  • ☰ 7.4. Common Causes of Training Instability
                                                                                    • ☰ 7.5. Interpreting Learning Curves
                                                                                      • ☰ 7.6. Practical Tips for Faster Iteration
                                                                                      • ☰ 8. Feedforward Networks for Structured Data
                                                                                        • ☰ 8.1. Feature Scaling and Normalization
                                                                                          • ☰ 8.2. Embeddings for Categorical Features
                                                                                            • ☰ 8.3. Multi-Layer Perceptrons in PyTorch
                                                                                              • ☰ 8.4. Choosing Hidden Sizes and Depth
                                                                                                • ☰ 8.5. Evaluation Metrics for Classification and Regression
                                                                                                  • ☰ 8.6. Model Export and Inference Basics
                                                                                                  • ☰ 9. Convolutional Neural Networks for Images
                                                                                                    • ☰ 9.1. Convolutions, Padding, and Stride
                                                                                                      • ☰ 9.2. Pooling and Feature Maps
                                                                                                        • ☰ 9.3. Building a Basic CNN in PyTorch
                                                                                                          • ☰ 9.4. Data Augmentation for Vision
                                                                                                            • ☰ 9.5. Transfer Learning with Pretrained Models
                                                                                                              • ☰ 9.6. Fine-Tuning and Freezing Layers
                                                                                                              • ☰ 10. Recurrent and Sequence Models
                                                                                                                • ☰ 10.1. Sequences, Tokens, and Padding
                                                                                                                  • ☰ 10.2. RNNs, LSTMs, and GRUs
                                                                                                                    • ☰ 10.3. Handling Variable-Length Sequences
                                                                                                                      • ☰ 10.4. Teacher Forcing and Sequence Losses
                                                                                                                        • ☰ 10.5. Basic Text Classification Pipeline
                                                                                                                          • ☰ 10.6. Simple Sequence-to-Sequence Overview
                                                                                                                          • ☰ 11. Attention and Transformers Fundamentals
                                                                                                                            • ☰ 11.1. Why Attention Helps
                                                                                                                              • ☰ 11.2. Self-Attention Intuition
                                                                                                                                • ☰ 11.3. Transformer Building Blocks
                                                                                                                                  • ☰ 11.4. Using Transformer Models in PyTorch
                                                                                                                                    • ☰ 11.5. Fine-Tuning for Text Classification
                                                                                                                                      • ☰ 11.6. Managing Tokenizers and Batching
                                                                                                                                      • ☰ 12. Model Evaluation and Experiment Management
                                                                                                                                        • ☰ 12.1. Train, Validation, and Test Best Practices
                                                                                                                                          • ☰ 12.2. Cross-Validation Basics
                                                                                                                                            • ☰ 12.3. Calibration and Thresholding
                                                                                                                                              • ☰ 12.4. Error Analysis and Confusion Matrices
                                                                                                                                                • ☰ 12.5. Experiment Tracking Concepts
                                                                                                                                                  • ☰ 12.6. Comparing Models Fairly
                                                                                                                                                  • ☰ 13. Saving, Loading, and Deployment Readiness
                                                                                                                                                    • ☰ 13.1. Saving Model Weights and Full Checkpoints
                                                                                                                                                      • ☰ 13.2. Loading for Inference and Resuming Training
                                                                                                                                                        • ☰ 13.3. TorchScript and Tracing Basics
                                                                                                                                                          • ☰ 13.4. Exporting to ONNX Overview
                                                                                                                                                            • ☰ 13.5. Batch Inference and Latency Considerations
                                                                                                                                                              • ☰ 13.6. Reproducible Packaging of Code and Weights
                                                                                                                                                              • ☰ 14. Performance and Scaling Basics
                                                                                                                                                                • ☰ 14.1. Profiling and Bottleneck Identification
                                                                                                                                                                  • ☰ 14.2. DataLoader Optimization
                                                                                                                                                                    • ☰ 14.3. GPU Utilization and Memory Management
                                                                                                                                                                      • ☰ 14.4. Gradient Accumulation for Large Batches
                                                                                                                                                                        • ☰ 14.5. Distributed Training Concepts Overview
                                                                                                                                                                          • ☰ 14.6. Practical Guidelines for Faster Training
                                                                                                                                                                          • ☰ 15. Best Practices and Next Steps
                                                                                                                                                                            • ☰ 15.1. Choosing Hyperparameters Systematically
                                                                                                                                                                              • ☰ 15.2. Regularization and Generalization Toolkit
                                                                                                                                                                                • ☰ 15.3. Responsible Use and Dataset Considerations
                                                                                                                                                                                  • ☰ 15.4. Reading PyTorch Documentation Effectively
                                                                                                                                                                                    • ☰ 15.5. Suggested Projects for Beginners
                                                                                                                                                                                      • ☰ 15.6. Pathways to Advanced Topics