Table of Contents
Why metrics matter beyond the loss
During training you minimize a loss function because it is convenient to optimize with gradients. For evaluation you usually want metrics that match the real goal, are easy to interpret, and can be compared across models and datasets. A useful habit is to log both the loss and one or two task specific metrics on the validation set, then use the metric to choose the best checkpoint.
Do not pick your final model based on training metrics. Use validation metrics for model selection and a test set metric only once at the end.
Classification metrics
For classification, start by being explicit about the prediction type your metric expects: class probabilities, class labels, or scores. Many mistakes come from mixing these up.
Accuracy and error rate
Accuracy is the fraction of correct predictions. With $N$ examples and predicted labels $\hat{y}_i$ and true labels $y_i$,
$$
\text{Accuracy} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}(\hat{y}_i = y_i).
$$
Error rate is $1 - \text{Accuracy}$. Accuracy is most informative when classes are balanced and mistakes have similar cost.
Accuracy can be misleading on imbalanced data. If 95 percent of samples are class 0, a model that always predicts class 0 gets 95 percent accuracy while being useless.
In PyTorch, for multi class classification you typically convert logits to labels with argmax(dim=1) and compare to integer targets.
Confusion matrix, precision, recall, and F1
A confusion matrix counts how predictions fall into true classes. For binary classification, define true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Then
$$
\text{Precision} = \frac{TP}{TP + FP},
\quad
\text{Recall} = \frac{TP}{TP + FN},
$$
and the F1 score is the harmonic mean:
$$
F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}.
$$
Precision answers, when the model predicts positive, how often is it correct. Recall answers, of all actual positives, how many did it find. F1 is useful when you want a single number that balances the two.
When moving beyond binary classification, you can compute precision, recall, and F1 per class by treating each class as positive versus the rest, then combine them.
Always state which averaging you use for multi class and multi label metrics. The same model can look better or worse depending on the averaging choice.
Micro, macro, and weighted averaging
For $K$ classes, you often report one number:
Micro averaging aggregates TP, FP, FN over all classes first, then computes precision and recall. This tends to favor frequent classes.
Macro averaging computes the metric per class then averages them equally. This treats rare classes as equally important.
Weighted macro is like macro but weights each class metric by its support, which is the number of true examples in that class.
Choose macro if you care about performance on minority classes, choose micro if you care about overall per example performance.
ROC AUC and PR AUC
Some metrics depend on a decision threshold, commonly 0.5 for binary probabilities. AUC metrics summarize performance across all thresholds.
ROC AUC uses the ROC curve, which plots true positive rate versus false positive rate. PR AUC uses the precision recall curve. In imbalanced settings, PR AUC is often more informative because it focuses on the positive class quality.
These metrics require scores, not hard labels. For binary classification, use predicted probabilities or logits for the positive class. For multi class, you can compute one versus rest AUC per class and average, but be explicit.
Do not compute AUC from argmax labels. AUC needs continuous scores so it can vary the threshold.
Log loss and top-k accuracy
Even when you train with cross entropy, it can be helpful to evaluate negative log likelihood as a metric to compare probabilistic quality.
For tasks with many classes, top k accuracy measures whether the true class is in the model’s top k predicted classes. This is common in large label problems.
Regression metrics
For regression, metrics compare predicted values $\hat{y}_i$ to targets $y_i$. The choice depends on how you want to penalize errors and whether outliers matter.
MAE and MSE, plus RMSE
Mean absolute error is
$$
\text{MAE} = \frac{1}{N}\sum_{i=1}^N |y_i - \hat{y}_i|.
$$
Mean squared error is
$$
\text{MSE} = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2.
$$
Root mean squared error is $\text{RMSE} = \sqrt{\text{MSE}}$, which brings units back to the original target scale. MSE and RMSE penalize large errors more strongly than MAE.
If your target has a physical unit, prefer MAE or RMSE for reporting because they share the target’s units. MSE is in squared units and can be harder to interpret.
$R^2$ coefficient of determination
$R^2$ measures how much variance your model explains compared to predicting the mean target. With $\bar{y}$ the mean of targets,
$$
R^2 = 1 - \frac{\sum_{i=1}^N (y_i - \hat{y}_i)^2}{\sum_{i=1}^N (y_i - \bar{y})^2}.
$$
An $R^2$ of 1 is perfect. An $R^2$ of 0 matches predicting the mean. It can be negative if the model is worse than predicting the mean.
MAPE and its caveats
Mean absolute percentage error is
$$
\text{MAPE} = \frac{100}{N}\sum_{i=1}^N \left|\frac{y_i - \hat{y}_i}{y_i}\right|.
$$
MAPE is easy to interpret as a percent error but becomes problematic when $y_i$ can be zero or very close to zero, and it can overemphasize errors on small targets.
Avoid MAPE when targets can be zero or near zero unless you have a clear strategy for handling division by small numbers.
Implementing metrics correctly in a PyTorch workflow
In evaluation you usually disable gradient tracking, move outputs and targets to the same device, and aggregate over batches.
For classification, be careful about the model output type. nn.CrossEntropyLoss expects logits of shape (batch, num_classes) and targets as integer class indices of shape (batch,). For metrics, you often want predicted labels from logits.argmax(dim=1) and compare to targets, or predicted probabilities from softmax when computing threshold based metrics or AUC.
For binary classification with a single logit output, a common pattern is to use torch.sigmoid(logits) to get probabilities, then choose a threshold to convert to labels. Keep the probabilities for PR AUC or ROC AUC style evaluation.
For regression, keep predictions and targets as floating point tensors with the same shape, often (batch,) or (batch, 1), and be consistent about squeezing dimensions before computing MAE or MSE.
Always verify shapes and dtypes when computing metrics. A silent broadcast or an unintended extra dimension can produce a plausible number that is wrong.
Choosing metrics for structured data problems
For structured data classification, class imbalance is common. Accuracy alone is rarely enough, so include macro F1 or per class recall, plus a confusion matrix during error analysis.
For structured data regression, MAE or RMSE is usually the best first metric. Add $R^2$ if stakeholders expect it, but remember it is relative to the variance in the dataset and can change with the evaluation split.
The most practical setup is to select one primary metric for early stopping and model comparison, and log a small set of secondary metrics for insight into failure modes.