Table of Contents
Shape Mismatches and Silent Broadcasting
Many PyTorch errors come from tensors having shapes that you did not intend. Some shape problems crash immediately, but others silently broadcast and produce a result that looks valid while being wrong.
A common example is mixing a batch of predictions shaped (N, 1) with targets shaped (N,). Depending on the operation, PyTorch may broadcast one of them and compute an (N, N) result, which is almost never what you want. The safest habit is to print shapes often and make your intended shapes explicit with view, reshape, unsqueeze, or squeeze at the point where the meaning is clear.
Rule: Before and after every nontrivial operation in early debugging, confirm tensor.shape. If two tensors are meant to align elementwise, their shapes should match exactly, or you should be able to state precisely which dimension is being broadcast and why.
When you do want broadcasting, it helps to make it obvious. For example, turning a vector of per feature scales with shape (D,) into (1, D) via unsqueeze(0) makes it clear you are broadcasting across the batch dimension.
Confusing `view` and `reshape`, and Non Contiguous Tensors
Beginners often use view to reshape tensors and get an error about contiguity. Many PyTorch operations return a tensor that is a view into the same storage with a different stride pattern. Such tensors can be non contiguous, meaning the memory layout does not match what view requires.
If x.view(...) fails, x.reshape(...) often works because it will return a view when possible and otherwise make a copy. If you specifically need a view for performance or memory reasons, you can call x = x.contiguous() before view, but do that intentionally since it may allocate memory.
Rule: Prefer reshape during debugging and learning. Use view when you know the tensor is contiguous, or after an explicit contiguous() that you chose for a reason.
A related pitfall is flattening. If you want to keep the batch dimension and flatten the rest, be explicit about it. The pattern is conceptually “flatten from dimension 1 onward,” which avoids accidentally collapsing the batch into the feature dimension.
Dtype Surprises and Integer Division
A tensor’s dtype affects both correctness and gradients. If your tensor is integer typed, many operations will produce integer outputs and gradients will not make sense, since autograd is designed for floating point computations. Another frequent issue is creating tensors from Python integers without specifying dtype, then later combining them with float tensors and getting unexpected type promotion behavior.
Loss functions and neural network layers almost always expect floating point inputs. Targets for classification losses are often integer class indices, while predictions are floating point logits. Confusing these can lead to cryptic errors or worse, silent mis training.
Rule: Model inputs and parameters should almost always be floating point, typically torch.float32. Ensure your features are float tensors before passing them into a network.
Also be careful when computing ratios or means from integer tensors. If you accidentally keep them as integers, you can lose information. A safe habit is to cast to float before such computations when you know the quantity is continuous.
Device Mismatches, CPU vs GPU
A very common runtime error is mixing tensors on different devices, such as adding a CPU tensor to a CUDA tensor. This often happens when you create a new tensor inside the training loop with torch.tensor(...) and forget to place it on the same device as the rest of the computation.
A reliable debugging approach is to check tensor.device for every input to the failing operation. When creating new tensors meant to interact with an existing tensor x, prefer factory methods that inherit device and dtype, such as creating values “like x” rather than from scratch. If you do create from scratch, pass device=... and dtype=... explicitly.
Rule: Any tensor created inside the forward or training loop that participates in computation should be created on the same device as the model inputs and parameters.
In Place Operations and Autograd Errors
PyTorch can throw errors like “one of the variables needed for gradient computation has been modified by an in place operation.” This happens when autograd needs an earlier value, but you overwrote it.
In place operations are easy to write by accident, for example using methods that end with an underscore, or using operators like += on tensors that require gradients. In place ops are sometimes safe, but when debugging, it is usually best to avoid them until everything works.
Rule: If you see an autograd error mentioning an in place operation, remove in place updates first. Replace x += y with x = x + y, and avoid methods ending with _ during debugging.
Activation functions can also be in place, such as nn.ReLU(inplace=True). If you run into gradient issues, temporarily set them to not be in place to rule that out.
Accidental Detaching, `.item()`, and Breaking the Graph
You can unintentionally stop gradients by detaching tensors. Calling tensor.detach() deliberately removes it from the graph. Converting a tensor to a Python number with .item() also breaks the graph. This is fine for logging, but not for computations that should remain differentiable.
Another subtle source is using with torch.no_grad(): in the wrong place, such as wrapping part of the forward pass during training.
Rule: Use .item() only for logging scalars. Never use .item() to feed a value back into a differentiable computation.
If training does not improve and gradients seem missing, search for detach, no_grad, and .item() around the code where the loss is computed.
NaNs, Infs, and Numerical Instability Checks
NaNs and infinities can appear due to invalid operations like division by zero, taking log(0), or exploding values. When this happens, the loss can become NaN and gradients become unusable.
When debugging, check tensors for finiteness at critical points such as model outputs, loss, and gradients. If you detect NaNs, locate where they first appear by checking intermediate values in the forward pass. This is often faster than guessing.
Rule: If the loss becomes NaN, stop and locate the first tensor in the forward pass that contains non finite values. Fixing the source is more effective than adding random clamps everywhere.
Confusing Batch and Feature Dimensions
Mixing up dimensions is easy, especially when switching between data types like tabular data, images, and sequences. Many layers assume a specific layout. For example, a linear layer expects (N, D) and a common mental error is to provide (D, N) because the math is written that way on paper.
A simple habit helps: decide early that the first dimension is batch, and consistently treat it that way throughout your project. When you transform data, verify that the batch dimension stayed in front.
Rule: Keep the batch dimension first unless you have a clear reason not to. If a layer expects batch first input, enforce it at the dataset and dataloader boundary.
Debugging Workflow: Minimal, Repeatable, Local
When something breaks, reduce the problem. Run a single forward pass with a tiny batch. Print shapes, dtypes, devices, and a small summary of value ranges like min and max. Confirm the loss is a scalar. Then run backward once and confirm gradients exist and are finite.
If an error is intermittent, it often depends on data. Use a fixed seed and a fixed mini batch to reproduce. If the issue disappears when you change batch size, it often indicates a broadcasting bug or a shape edge case.
A practical local checklist is to verify shape, dtype, device, and requires_grad for the tensors involved in the failing line, then work outward.
Rule: Debug on the smallest possible case that still reproduces the bug. One batch, one forward pass, one backward pass is often enough to isolate tensor issues.