12.4. Error Analysis and Confusion Matrices

Why error analysis matters after you have metrics

Overall metrics like accuracy, F1, or RMSE tell you how well a model performs on average, but they rarely tell you why it fails. Error analysis is the process of inspecting mistakes to discover patterns you can act on, such as mislabeled data, a confusing pair of classes, a missing feature, or a threshold choice that does not match the goal. The aim is not to collect interesting anecdotes, it is to find repeatable failure modes that suggest concrete next steps in data, modeling, or evaluation.

Confusion matrix fundamentals

A confusion matrix summarizes how often each true class is predicted as each class. For a classification problem with $K$ classes, the confusion matrix is a $K \times K$ table where entry $C_{i,j}$ counts examples whose true label is class $i$ and whose predicted label is class $j$. The main diagonal represents correct predictions, off diagonal cells represent confusions.

In binary classification, the confusion matrix is often described with four counts. True positives are cases where the true label is positive and the prediction is positive. True negatives are true negative and predicted negative. False positives are true negative but predicted positive, these are also called type I errors. False negatives are true positive but predicted negative, these are also called type II errors. Many metrics can be derived from these counts, but the confusion matrix itself is useful because it localizes errors to specific classes and directions.

Computing a confusion matrix in PyTorch

You can build a confusion matrix directly from tensors. Assume you have integer class labels in y_true and predicted labels in y_pred, both shaped (N,), with values in 0..K-1. A simple and fast approach uses a flattened index.

import torch
def confusion_matrix(y_true: torch.Tensor, y_pred: torch.Tensor, num_classes: int) -> torch.Tensor:
    y_true = y_true.to(torch.int64).view(-1)
    y_pred = y_pred.to(torch.int64).view(-1)
    idx = y_true * num_classes + y_pred
    cm = torch.bincount(idx, minlength=num_classes * num_classes)
    return cm.view(num_classes, num_classes)
# Example:
# y_true: tensor([0, 0, 1, 2])
# y_pred: tensor([0, 2, 1, 1])
# num_classes = 3
# cm = [[1,0,1],
#       [0,1,0],
#       [0,1,0]]

If your model outputs logits, you typically convert to predicted classes with argmax for multi class classification.

logits = model(x)                 # (N, K)
y_pred = logits.argmax(dim=1)     # (N,)
cm = confusion_matrix(y_true, y_pred, num_classes=K)

For binary classification with a single logit per example, you typically apply a sigmoid and threshold.

logits = model(x).view(-1)                # (N,)
probs = torch.sigmoid(logits)             # (N,)
y_pred = (probs >= 0.5).to(torch.int64)   # (N,)
cm = confusion_matrix(y_true, y_pred, num_classes=2)

Normalized confusion matrices for class imbalance

Raw counts can be misleading when classes are imbalanced, because large classes dominate the totals. A common remedy is to normalize by row so each row sums to 1. Row normalized values approximate per class error distributions, answering “given the true class is $i$, what do we predict?”

cm = confusion_matrix(y_true, y_pred, num_classes=K).to(torch.float32)
row_sums = cm.sum(dim=1, keepdim=True).clamp_min(1.0)
cm_row_norm = cm / row_sums

Column normalization is also used when you want to interpret “given we predicted class $j$, what are the true classes?” Choose the normalization that matches the question you are asking.

Reading the confusion matrix to find actionable issues

A useful workflow is to start with the largest off diagonal cells. Those indicate the most common confusions. If class A is often predicted as class B, you can ask whether the classes are truly separable, whether the labels are inconsistent, or whether the model lacks relevant input signals. Symmetric confusions, where A is mistaken for B and B for A, often indicate that the two categories overlap or are hard to distinguish in the available data. One directional confusions, where A becomes B but not vice versa, often indicate thresholding effects, class imbalance, or a feature that is present in one class but absent in the other.

For multi class problems, pay attention to classes that are rarely correctly classified, even if overall accuracy looks good. A low diagonal entry for a class can signal insufficient training data for that class, label noise, or a mismatch between training and evaluation conditions.

Practical error analysis loop: from matrix to examples

The confusion matrix tells you where to look, but actual examples tell you what is happening. After identifying a problematic cell $(i, j)$, collect a small sample of instances where y_true == i and y_pred == j and inspect them. In image tasks, you look at the images and augmentation effects. In text tasks, you read the text and check tokenization artifacts. In tabular tasks, you examine feature values, missingness patterns, and outliers.

A practical approach is to sort mistakes by model confidence. High confidence mistakes are often the most informative because they can indicate label errors, dataset leakage issues, or systematic bias. For multi class classification, you can use softmax probabilities, and for binary classification you can use sigmoid probabilities.

with torch.no_grad():
    logits = model(x)
    probs = torch.softmax(logits, dim=1)
    conf, y_pred = probs.max(dim=1)
mask = (y_true == i) & (y_pred == j)
idx = torch.where(mask)[0]
idx = idx[conf[idx].argsort(descending=True)]  # highest-confidence mistakes first

Common root causes you can uncover

Some failure patterns point to data problems. If the inspected examples look correctly predicted but labeled differently, you may have label noise or ambiguous labeling rules. If mistakes cluster around certain sources, such as one camera type, one demographic group, or one text domain, you may have dataset shift within your evaluation set.

Other patterns suggest modeling issues. If the model confuses classes that differ by fine detail, it may need higher resolution inputs, a different architecture, or different augmentations. If many errors happen on borderline examples with low confidence, the model may be under trained, under parameterized, or the problem may be intrinsically ambiguous.

Threshold and decision policy issues show up clearly in binary confusion matrices. A high false positive rate may be unacceptable in some applications, while a high false negative rate may be unacceptable in others. In those cases, error analysis should be paired with a threshold choice that matches the cost of each error type, rather than relying on a default 0.5 threshold.

Reporting confusion matrices correctly

When you present a confusion matrix, include the label order, whether it is normalized, and the split it was computed on. For multi class problems, also include support per class, meaning the number of true examples in each row. If you compare two models, use the same label order and the same normalization so the differences are interpretable.

Finally, treat the confusion matrix as a guide for the next iteration. The best outcome of error analysis is a short list of targeted actions, such as collecting more data for specific classes, revising label definitions, adding features, adjusting preprocessing, or changing the thresholding policy.

Views: 74