Table of Contents
What This Section Is For
Training deep learning models is an iterative process, and debugging is the skill that keeps that iteration productive. This section focuses on turning confusing outcomes into concrete checks you can run, so you can answer questions like, “Is my model broken, is my data wrong, or is my setup unstable?” The goal is not to memorize every possible failure mode, but to build a reliable routine that narrows the problem quickly and prevents repeating the same mistakes.
A Practical Debugging Mindset
Most training problems fall into a few categories: incorrect tensor shapes, wrong targets or label encoding, numerical issues such as NaNs and infinities, gradients that are zero or explode, a mismatch between model output and loss function expectations, or a data pipeline that does not produce what you think it produces. Debugging works best when you treat your code as a system with checkpoints. You want to validate each stage, from raw batch to model outputs to loss to gradients, before you spend time tuning hyperparameters.
The core idea is to reduce uncertainty. Instead of changing many things at once, make one change, measure one outcome, and keep notes. If you cannot reproduce an issue, you cannot reliably fix it, so your first step is often to make runs deterministic enough to compare behavior across changes, using the reproducibility tools introduced earlier in the course.
A Standard Triage Workflow
When training behaves unexpectedly, a good default workflow is to start with the cheapest checks and move toward more expensive ones. First, confirm the data batch looks correct and stable across iterations. Then confirm the model forward pass produces outputs with the shape, dtype, and value range you expect. Next, verify the loss is finite and decreases at least on a tiny sample. Finally, inspect gradients and parameter updates.
If you are stuck, do not tune hyperparameters first. Confirm correctness first: batch contents, model outputs, loss inputs, loss finiteness, and gradients.
Types of Failure You Will See Often
Many failures are “silent,” meaning your code runs without errors but trains poorly. A common example is feeding class labels with the wrong dtype into a classification loss, or applying a softmax twice. Another is accidentally shuffling features and labels out of alignment. You also often see learning curves that look plausible but represent the wrong objective because metrics are computed incorrectly or on the wrong split.
Other failures are “loud,” meaning they produce exceptions or NaNs. These usually come from invalid operations such as taking $\log(0)$, dividing by zero during normalization, numerical overflow in exponentials, or unstable loss configurations.
Observability, Print Debugging, and Minimal Instrumentation
For beginners, the simplest tool is still printing, but printing everything is overwhelming and slow. Prefer printing summaries that answer specific questions: batch shapes, minimum and maximum values, and whether tensors contain NaNs or infinities. A small amount of instrumentation can be permanent, such as assertions that check expected shapes or that loss remains finite.
A useful habit is to confirm both the type and the meaning of a tensor. A tensor might have the “right” shape but contain logits when you think it contains probabilities, or contain normalized values when you think it contains raw pixels.
Always know whether your model outputs are logits or probabilities, and choose losses and metrics accordingly. Mismatching these often trains without crashing but produces misleading results.
Debugging Is Not Only Code, It Is Also Process
Even with perfect code, training can fail because your expectations are wrong. If the dataset is tiny, noisy, or mislabeled, the model may not improve much. If the validation set is too small, metrics can jump around and look unstable. If you evaluate in the wrong mode, results can differ. If you compare experiments unfairly, you may chase phantom improvements.
A strong process is to establish a baseline that you can always return to. That baseline might be a simple model that trains quickly and behaves predictably. When a more complex model fails, you can compare against the baseline to see whether the failure is architectural or systemic.
What You Will Learn in the Subchapters
The rest of this section breaks debugging into concrete, repeatable checks. You will learn how to verify tensor shapes and detect NaNs early, how to inspect gradients and recognize vanishing or exploding gradients, how to run sanity checks such as intentionally overfitting a tiny batch, how to identify common causes of training instability, how to interpret learning curves without fooling yourself, and how to iterate faster by structuring experiments and reducing turnaround time.
By the end of this section, you should be able to take a model that “does not learn” and methodically determine whether the issue is data, model definition, loss configuration, optimization dynamics, or evaluation methodology.