7.4 Common Causes of Training Instability

What Training Instability Looks Like

Training becomes unstable when your loss or metrics behave erratically instead of improving in a mostly steady trend. Typical symptoms include the loss suddenly exploding to very large values, the loss becoming NaN or Inf, gradients spiking unpredictably, accuracy jumping up and down without settling, or the model appearing to learn for a few steps and then collapsing. Instability is not the same as slow learning, it is the situation where optimization stops being well behaved and you cannot trust the direction of updates.

Learning Rate Problems

The most common cause is a learning rate that is too high for the model, the optimizer, the batch size, or the current phase of training. When the learning rate is too large, parameter updates overshoot and bounce around the landscape, which often shows up as oscillating loss and occasional divergence. A learning rate that is too low rarely causes true instability, but it can interact with other issues, for example mixed precision scaling or Adam epsilon choices, to produce noisy behavior.

Bad Input Scale and Missing Normalization

Unstable training often comes from inputs that are on wildly different scales, or from targets that are not scaled appropriately for the chosen loss. Large magnitude inputs can push activations into extreme ranges, which can produce large gradients and overflow in floating point arithmetic. This is especially common for tabular features that mix counts, prices, and binary indicators without normalization, or for regression targets with huge units.

A related issue is using an output activation that mismatches the target scale. For example, constraining outputs to a small range while targets are unbounded forces the model to compensate with extreme internal weights, which makes training numerically fragile.

Loss Function and Output Mismatch

Instability can come from pairing the wrong loss with the model output. A classic case is applying a sigmoid in the model and also using a loss that expects raw logits, or using a loss that expects probabilities but feeding it logits. These mismatches can create saturated activations, near zero gradients in some regions, and extremely steep gradients in others.

Another common source is reduction and scaling. If you accidentally sum a loss over many elements when you intended to average, the gradient magnitude can grow with batch size or sequence length, making the effective step size much larger than expected.

Exploding Gradients and Deep or Recurrent Models

Some architectures are more prone to instability, especially deep networks without normalization, recurrent models on long sequences, and models with poorly conditioned layers. Exploding gradients occur when backpropagated derivatives grow exponentially through the network, leading to very large parameter updates and loss divergence. While gradient clipping is a common mitigation, instability here usually also indicates that learning rate, initialization, normalization, or sequence length handling needs attention.

Numerical Issues and NaNs

NaNs and Infs can appear due to invalid operations such as taking $\log(0)$, dividing by zero, overflow in exponentials, or square roots of negative values. They can also appear from extreme activations that exceed the representable range in float16 or even float32. Softmax and log softmax are frequent hotspots if implemented naively, which is why stable library functions are typically used.

Even if your code avoids explicit invalid math, NaNs can arise from accumulating very large values over time, for example in poorly scaled losses, or from unstable variance computations.

Mixed Precision and Underflow or Overflow

When using mixed precision, instability can come from underflow in float16 gradients or overflow in activations. Automatic mixed precision and gradient scaling reduce this risk, but they do not eliminate it, especially when the model produces unusually large intermediate values or when the loss scale fluctuates aggressively.

If instability appears only when enabling mixed precision, suspect numerical range issues before suspecting the model logic itself. In practice, a lower learning rate, ensuring gradient scaling is enabled, and keeping certain operations in float32 often resolves it.

Batch Size Effects and Noisy Gradients

Very small batch sizes can make gradients highly noisy. Noise by itself is not always bad, but it can interact with a high learning rate or a sharp loss surface to produce chaotic updates. Conversely, very large batch sizes can make optimization more sensitive to learning rate choices, since the gradient becomes more consistent and steps can become too aggressive if the learning rate is not tuned accordingly.

If training becomes unstable when you change batch size, interpret it as an effective change in optimization behavior, not as a random fluke.

Data Problems That Masquerade as Optimization Issues

Corrupted data, incorrect labels, inconsistent preprocessing between training and validation, or train time augmentations that produce out of range values can all look like instability. Examples include images with unexpected value ranges, token IDs outside the vocabulary, or normalization applied twice. Another subtle case is class imbalance combined with an unsuitable loss setup, where rare classes cause occasional large gradients when they appear.

If instability appears only on certain batches, it often points to a data issue in those batches rather than a global hyperparameter problem.

Weight Initialization and Saturating Activations

Poor initialization can place activations in saturated regimes, especially with sigmoid or tanh, where gradients become extremely small in some regions and occasionally large when the model escapes saturation. Even with ReLU like activations, bad initialization can create layers that output extremely large values, producing large gradients and numerical overflow.

While many PyTorch layers have reasonable defaults, instability is more likely when you build custom layers, manually initialize weights, or stack many layers without normalization.

Regularization and Constraints Applied Too Aggressively

Regularization can destabilize training when applied with the wrong magnitude. Excessive weight decay can fight against the gradient signal and cause erratic progress, especially with adaptive optimizers if the decay is not configured correctly. Dropout usually does not cause divergence, but very high dropout rates can increase gradient variance and create noisy loss curves that resemble instability.

Constraints like hard clamping of weights or outputs can also introduce sharp non smooth behavior that makes optimization brittle.

Putting It Together as a Quick Triage

When training is unstable, prioritize the highest probability causes in a tight loop. First reduce the learning rate, then confirm input and target scaling, then verify loss and output compatibility, then check for NaNs and numerical hotspots, then consider gradient clipping and architecture specific stabilization. If instability is batch dependent, investigate the data pipeline and preprocessing for outliers or invalid values.