Table of Contents
What Learning Curves Are Telling You
A learning curve is the history of how your model behaves over time, usually plotted as a function of training steps or epochs. The most common curves are training loss and validation loss, and sometimes training and validation metrics such as accuracy, F1, or mean absolute error. Interpreting these curves is about diagnosing what is likely happening inside training, then choosing a targeted change rather than guessing.
A key idea is that loss curves and metric curves do not mean the same thing. Loss is the quantity you optimize, metrics are often what you care about. It is normal for loss to keep improving while a metric barely changes, especially when the model is getting more confident on already correct examples.
Plot the Right Things and Align Them Correctly
To interpret curves reliably, you must ensure you are plotting comparable values. Training loss is often averaged over mini batches during the epoch, while validation loss is computed at the end of the epoch. If you compare a noisy per batch training curve against a smooth per epoch validation curve, you can misread normal noise as instability. A practical approach is to log both as epoch averages, or log both per step, but be consistent.
Also make sure training and validation are computed in the correct modes. Validation must be computed with model.eval() and inside torch.no_grad(). Otherwise dropout and batch normalization behavior can distort validation curves and make them look artificially noisy or too optimistic.
Validation curves are only meaningful if validation is run with model.eval() and torch.no_grad(). If you forget, you can misdiagnose overfitting, instability, or “random” validation behavior.
The Healthy Baseline Pattern
A common healthy pattern early in training is that training loss decreases quickly, validation loss decreases as well, then both improvements slow down. Training loss is typically lower than validation loss because the model is optimized on the training data.
When you see this pattern, most “fixes” are about efficiency and fine tuning, such as adjusting learning rate schedules or training longer, rather than repairing something fundamentally broken.
Classic Patterns and What They Usually Mean
Overfitting Pattern
Overfitting often looks like training loss continues to go down while validation loss stops improving and starts rising. Validation metrics may plateau or degrade while training metrics keep improving. This usually means the model is learning training specific details that do not generalize.
Typical responses include increasing regularization, reducing model capacity, using more data or augmentation, and using early stopping based on validation loss. The best action depends on which tool you are already using, this chapter is about recognizing the pattern rather than choosing the full regularization strategy.
If validation loss rises while training loss falls, do not keep training blindly. Capture the best checkpoint at the minimum validation loss, and investigate overfitting rather than assuming more epochs will fix it.
Underfitting Pattern
Underfitting tends to show training loss that decreases slowly and then levels off at a relatively high value, with validation loss close to training loss and also high. Metrics may be poor on both. The gap between training and validation is small because the model is not fitting even the training set well.
This often indicates insufficient model capacity, too much regularization, or optimization that is not making progress. It can also happen when inputs are not informative or labels are noisy, but start by checking model and training configuration.
High Variance, Noisy Curves
If training loss is extremely noisy from step to step, or validation metrics swing wildly between epochs, common causes are very small batch size, a learning rate that is too high, an overly small validation set, or randomness such as dropout affecting evaluation because of incorrect mode.
Noise is not automatically bad. Stochastic training is expected to wiggle. The interpretation hinges on whether the overall trend improves. Smoothing with a moving average can help you see the trend, but you should still keep the raw curve available to spot spikes and instability.
Divergence and Exploding Loss
If loss suddenly shoots up to very large values, becomes inf, or becomes NaN, training is unstable. This is often caused by too large a learning rate, numerical issues, exploding gradients, or invalid operations in the forward pass.
One sign is a loss curve that decreases a bit, then abruptly becomes undefined. Another is a sawtooth pattern of huge spikes that never settle. In these cases you should treat the curve as a symptom of a failure mode, not as “training longer” territory.
A loss that becomes NaN or inf is not a normal learning curve shape. Stop and debug immediately, log where it happens, and investigate learning rate, data validity, and numerical stability.
Curves That Look Too Good to Be True
If validation performance is surprisingly excellent very early, or validation loss is consistently lower than training loss by a large margin, suspect data leakage, overlap between train and validation sets, or a mismatch in how you compute the two losses. Another possibility is that training uses augmentations or dropout that make training harder, but the gap should still be believable.
A particularly suspicious pattern is a validation curve that improves dramatically while training barely changes, which can happen if validation accidentally uses training data or if the split is incorrect.
When Loss and Accuracy Disagree
For classification, you may see training loss decreasing while accuracy stays flat. That can happen because cross entropy loss depends on predicted probabilities, not just correct versus incorrect. The model can get more confident on already correct examples, lowering loss, without changing the number of correct predictions. Conversely, accuracy can increase while loss worsens if the model becomes less calibrated but still crosses the decision threshold on more examples.
This is why it is valuable to track both loss and at least one task metric, then interpret them together.
The Generalization Gap as a Diagnostic Signal
The difference between training and validation performance is often called the generalization gap. A small gap with bad performance suggests underfitting. A large gap suggests overfitting. A gap that grows over time is a warning sign for when to stop training and revert to the best validation checkpoint.
Do not over interpret small gaps. Measurement noise, small validation sets, and randomness can create apparent gaps. Look for consistent separation across multiple epochs.
Using Learning Curves to Choose a Next Experiment
Learning curves are most useful when they tell you what to change next. If both training and validation losses are decreasing steadily, the simplest next experiment is often to train longer or use a learning rate schedule that decays later. If training improves but validation worsens, prioritize regularization, data augmentation, or early stopping. If neither improves, investigate underfitting or optimization issues.
The key is to make one change at a time and compare curves fairly, using the same logging frequency, same split, and same evaluation procedure.
Only compare learning curves across runs if the train, validation split, preprocessing, metric computation, and logging cadence are consistent. Otherwise you can “prove” improvements that are just measurement differences.
A Minimal Checklist When Reading Curves
Before drawing conclusions from a curve, confirm that the axes are correct, that you are comparing the same units such as per epoch averages, that validation uses evaluation mode, and that the run is not corrupted by NaN events or data leakage. Once those basics are correct, the curve shapes become a reliable compass for whether you are underfitting, overfitting, unstable, or simply still learning.