Table of Contents
Probabilities vs Decisions
Many classifiers can output a score for each class, often turned into a probability with a sigmoid for binary classification or a softmax for multi class classification. These numbers look like confidence, but they are not guaranteed to be trustworthy as probabilities. Calibration is about whether predicted probabilities match real world frequencies. Thresholding is about how you convert probabilities or scores into decisions such as positive vs negative, or which class to choose.
A useful mental model is to separate two questions. First, how well can the model rank examples from most likely positive to least likely positive. Second, if the model says 0.8, does the event happen about 80 percent of the time. Ranking quality is closely tied to metrics like ROC AUC, while probability trustworthiness is calibration. A model can rank well but be poorly calibrated, or be well calibrated but not rank well.
What Calibration Means
In binary classification, a model is perfectly calibrated if among all examples where it predicts probability $p$, the true positive rate is also $p$. In practice you check this by grouping predictions into bins, for example 0.0 to 0.1, 0.1 to 0.2, and so on, then comparing average predicted probability to observed frequency in each bin.
Important rule: a higher accuracy does not imply better calibration, and a better calibration does not imply better accuracy. Treat calibration as a separate property you measure.
Calibration matters when you use probabilities to make decisions under uncertainty, such as risk scoring, triage, or any setting where different mistakes have different costs.
When You Need Thresholding
Thresholding is the step that turns a predicted probability $\hat{p}$ into a class label. For binary classification, the most common default is threshold $t = 0.5$, predicting positive when $\hat{p} \ge t$. This is only a default, it is rarely optimal unless classes are balanced and costs are symmetric.
The threshold controls the tradeoff between false positives and false negatives. Raising the threshold generally reduces false positives and increases false negatives, lowering it generally does the opposite. Your choice should reflect what matters for the task, which is why thresholds are often selected using the validation set, then reported on the test set once at the end.
Do not choose a threshold using the test set. Use the validation set for threshold selection, then lock it and evaluate once on the test set.
Cost Sensitive Threshold Selection
If you can quantify the cost of errors, you can derive a principled threshold. Suppose you predict positive when $\hat{p} \ge t$. Let $C_{FP}$ be the cost of a false positive and $C_{FN}$ be the cost of a false negative. Under standard assumptions, the cost minimizing decision rule is:
$$
\text{predict positive if } \hat{p} \ge \frac{C_{FP}}{C_{FP} + C_{FN}}.
$$
This formula is a clean starting point. In practice you might still tune around it because the costs are approximations and the model probabilities may be miscalibrated.
Key formula: if you know costs, use $t = \frac{C_{FP}}{C_{FP} + C_{FN}}$ as a principled threshold for binary decisions, assuming probabilities are calibrated.
Thresholding in Multi Class Problems
For multi class classification with softmax probabilities, the most common decision rule is argmax, choose the class with highest probability. Thresholding still appears in two common ways. One is abstention, only make a prediction if the top probability exceeds some threshold, otherwise return unknown. Another is one vs rest decisions, where you assign multiple labels or decide membership per class using per class thresholds.
Per class thresholds are often needed when classes are imbalanced or when different classes have different error costs. This becomes a validation tuning problem rather than a single global threshold.
How Calibration Interacts With Thresholding
Threshold selection assumes that a probability of 0.7 means the example truly has about 70 percent chance of being positive. If the model is overconfident or underconfident, the threshold that matches a desired operating point can shift. For example, if a model is systematically overconfident, a threshold of 0.8 may behave more like 0.6 in reality, producing more positives than expected.
Calibration also affects any decision based on probability targets, such as picking a threshold to achieve a target precision, or setting different actions for different risk bands. If probabilities are unreliable, those policies become unstable.
If you use predicted probabilities to set policies, risk tiers, or cost based decisions, measure calibration and consider calibrating the model before final threshold selection.
Practical PyTorch Workflow for Threshold Tuning
In PyTorch, thresholding is usually applied outside the model. Your model returns logits or probabilities, then evaluation code converts them into predicted labels. Keeping thresholding outside the model helps you change operating points without retraining.
A typical workflow is to run the model on the validation set, store the predicted probabilities and true labels, sweep thresholds from 0 to 1, and compute the metric you care about at each threshold. You then pick the best threshold under your chosen criterion, such as maximizing F1, achieving recall above a minimum, or minimizing an estimated cost. Once selected, you keep that threshold fixed for test evaluation and for deployment.
Common Pitfalls
A frequent mistake is to assume the default threshold of 0.5 is meaningful. Another is to compare two models at different thresholds without stating the operating point, which can make results misleading. It is also common to tune a threshold to maximize a metric like F1 and then claim that the resulting probability values are well calibrated, which is not true, thresholding does not fix probability quality.
Finally, remember that calibration and thresholding are validation time choices that can drift over time if the data distribution changes. In applied settings, you often need to monitor calibration and refresh thresholds as part of model maintenance.