Table of Contents
What to Track During Training
Training produces many numbers, but two categories matter most: the loss you optimize and the metrics you care about. Loss is the scalar your optimizer tries to minimize, like mean squared error for regression or cross entropy for classification. Metrics are usually human meaningful scores such as accuracy, precision, recall, or $R^2$. A key habit is to track both for both training and validation sets, because they answer different questions. Training values tell you whether the model can fit the data it sees, validation values tell you whether what it learned generalizes.
Always log loss for training and validation separately. If you only track training loss, you can miss overfitting. If you only track validation loss, you can miss optimization bugs.
Loss vs Metric, and Why They Differ
Loss is designed to be smooth and differentiable, so gradient descent can optimize it. Metrics are often not differentiable or are piecewise constant, like accuracy, so they can be poor optimization targets even if they are great for evaluation. It is common to see loss improve while a metric moves slowly, especially early in training. This is not automatically a problem.
For binary or multiclass classification, you might optimize cross entropy but report accuracy. For regression, you might optimize mean squared error but report mean absolute error to interpret average error in original units. The important idea is consistency: decide what you optimize, decide what you report, then track them the same way across experiments.
Computing Averages Correctly Over an Epoch
When you train with mini batches, each batch gives you a loss and possibly a metric. A common mistake is to average batch losses by taking the mean of the per batch losses without accounting for different batch sizes. This can bias your epoch numbers if the last batch is smaller or if you use uneven batches.
The safest pattern is to accumulate a weighted sum by batch size and divide by the total number of examples. If $L_i$ is the average loss returned for batch $i$ and $n_i$ is its batch size, then the dataset average loss is
$$
\bar{L} = \frac{\sum_i L_i \cdot n_i}{\sum_i n_i}.
$$
The same weighting principle applies to many metrics. For accuracy, you can accumulate correct predictions and divide by total examples.
Do not assume all batches are the same size. Compute epoch averages using total counts, not the number of batches.
Classification Metrics in Practice
Accuracy is straightforward, but you should compute it from discrete predictions, not from probabilities. For multiclass classification with logits of shape [batch, num_classes], you typically convert to predicted class indices with argmax. For binary classification with a single logit per example, you typically apply a threshold on the sigmoid probability, often $0.5$, but the threshold can change depending on the problem.
When you calculate classification metrics, make sure you are consistent about whether you are using logits or probabilities. Many PyTorch losses take logits directly, for example nn.CrossEntropyLoss expects raw logits and internally applies a log softmax, and nn.BCEWithLogitsLoss expects raw logits and internally applies a sigmoid.
If you use CrossEntropyLoss, do not apply softmax before the loss. If you use BCEWithLogitsLoss, do not apply sigmoid before the loss. Compute metrics separately from the loss using the appropriate conversions.
Regression Metrics in Practice
For regression, prediction and target have the same shape, often [batch, 1] or [batch]. The loss might be MSE, but you may also track MAE for interpretability. If your targets were normalized, remember that metrics computed in normalized space may be hard to interpret. In that case, you can compute metrics after unnormalizing, but keep the training loss in normalized space if that is how you trained.
A typical regression tracking setup is training loss as MSE, plus validation MAE in original units. This gives you an optimization signal and a user facing measure of error.
Training vs Validation Mode When Tracking
Metrics should be computed with the model in the correct mode. During training you call model.train(), which enables behaviors like dropout and updates running statistics in batch normalization. During validation you call model.eval(), which disables dropout and uses batch norm running estimates. If you forget this, validation metrics can look noisy or systematically worse for reasons unrelated to learning.
Validation should also avoid tracking gradients. Wrap validation in torch.no_grad() so it uses less memory and runs faster. This also reduces the chance of accidentally backpropagating through validation computations.
Use model.train() for training and model.eval() with torch.no_grad() for validation. Logging validation metrics in training mode can mislead you.
A Minimal Pattern for Tracking Scalars
A clean way to track training is to compute and store a few scalar values per epoch, such as training loss, validation loss, and one or two metrics. At the end of each epoch you print them and optionally append them to lists so you can plot later.
The typical flow is: initialize accumulators, loop over the DataLoader, update accumulators, compute epoch averages, then repeat for validation. Store results in a history dictionary keyed by metric name so you can compare runs easily.
Interpreting What You See
When tracking, you are looking for patterns more than individual values. If training loss decreases and validation loss also decreases, learning is progressing and generalization is improving. If training loss decreases but validation loss rises, you are likely overfitting. If neither training nor validation loss changes, you might have a learning rate issue, a bug in the training loop, or a model that is too small for the task.
Metrics may lag behind loss, especially accuracy in classification. Loss can improve because the model becomes more confident on examples it already gets right, which does not change accuracy but still indicates learning. That is why tracking both is valuable.
Judge progress using trends across multiple epochs. Single epoch spikes are common, especially with small datasets or small batch sizes.