12.3. Calibration and Thresholding
Table of Contents
Probabilities vs Decisions
Many classifiers can output a score for each class, often turned into a probability with a sigmoid for binary classification or a softmax for multi class classification. These numbers look like confidence, but they are not guaranteed to be trustworthy as probabilities. Calibration is about whether predicted probabilities match real world frequencies. Thresholding is about how you convert probabilities or scores into decisions such as positive vs negative, or which class to choose.
A useful mental model is to separate two questions. First, how well can the model rank examples from most likely positive to least likely positive. Second, if the model says 0.8, does the event happen about 80 percent of the time. Ranking quality is closely tied to metrics like ROC AUC, while probability trustworthiness is calibration. A model can rank well but be poorly calibrated, or be well calibrated but not rank well.
What Calibration Means
In binary classification, a model is perfectly calibrated if among all examples where it predicts probability $p$, the true positive rate is also $p$. In practice you check this by grouping predictions into bins, for example 0.0 to 0.1, 0.1 to 0.2, and so on, then comparing average predicted probability to observed frequency in each bin.
Important rule: a higher accuracy does not imply better calibration, and a better calibration does not imply better accuracy. Treat calibration as a separate property you measure.
Calibration matters when you use probabilities to make decisions under uncertainty, such as risk scoring, triage, or any setting where different mistakes have different costs.
When You Need Thresholding
Thresholding is the step that turns a predicted probability $\hat{p}$ into a class label. For binary classification, the most common default is threshold $t = 0.5$, predicting positive when $\hat{p} \ge t$. This is only a default, it is rarely optimal unless classes are balanced and costs are symmetric.
The threshold controls the tradeoff between false positives and false negatives. Raising the threshold generally reduces false positives and increases false negatives, lowering it generally does the opposite. Your choice should reflect what matters for the task, which is why thresholds are often selected using the validation set, then reported on the test set once at the end.
Do not choose a threshold using the test set. Use the validation set for threshold selection, then lock it and evaluate once on the test set.
Cost Sensitive Threshold Selection
If you can quantify the cost of errors, you can derive a principled threshold. Suppose you predict positive when $\hat{p} \ge t$. Let $C_{FP}$ be the cost of a false positive and $C_{FN}$ be the cost of a false negative. Under standard assumptions, the cost minimizing decision rule is:
$$
\text{predict positive if } \hat{p} \ge \frac{C_{FP}}{C_{FP} + C_{FN}}.
$$
This formula is a clean starting point. In practice you might still tune around it because the costs are approximations and the model probabilities may be miscalibrated.
Key formula: if you know costs, use $t = \frac{C_{FP}}{C_{FP} + C_{FN}}$ as a principled threshold for binary decisions, assuming probabilities are calibrated.
Thresholding in Multi Class Problems
For multi class classification with softmax probabilities, the most common decision rule is argmax, choose the class with highest probability. Thresholding still appears in two common ways. One is abstention, only make a prediction if the top probability exceeds some threshold, otherwise return unknown. Another is one vs rest decisions, where you assign multiple labels or decide membership per class using per class thresholds.
Per class thresholds are often needed when classes are imbalanced or when different classes have different error costs. This becomes a validation tuning problem rather than a single global threshold.
How Calibration Interacts With Thresholding
Threshold selection assumes that a probability of 0.7 means the example truly has about 70 percent chance of being positive. If the model is overconfident or underconfident, the threshold that matches a desired operating point can shift. For example, if a model is systematically overconfident, a threshold of 0.8 may behave more like 0.6 in reality, producing more positives than expected.
Calibration also affects any decision based on probability targets, such as picking a threshold to achieve a target precision, or setting different actions for different risk bands. If probabilities are unreliable, those policies become unstable.
If you use predicted probabilities to set policies, risk tiers, or cost based decisions, measure calibration and consider calibrating the model before final threshold selection.
Practical PyTorch Workflow for Threshold Tuning
In PyTorch, thresholding is usually applied outside the model. Your model returns logits or probabilities, then evaluation code converts them into predicted labels. Keeping thresholding outside the model helps you change operating points without retraining.
A typical workflow is to run the model on the validation set, store the predicted probabilities and true labels, sweep thresholds from 0 to 1, and compute the metric you care about at each threshold. You then pick the best threshold under your chosen criterion, such as maximizing F1, achieving recall above a minimum, or minimizing an estimated cost. Once selected, you keep that threshold fixed for test evaluation and for deployment.
Common Pitfalls
A frequent mistake is to assume the default threshold of 0.5 is meaningful. Another is to compare two models at different thresholds without stating the operating point, which can make results misleading. It is also common to tune a threshold to maximize a metric like F1 and then claim that the resulting probability values are well calibrated, which is not true, thresholding does not fix probability quality.
Finally, remember that calibration and thresholding are validation time choices that can drift over time if the data distribution changes. In applied settings, you often need to monitor calibration and refresh thresholds as part of model maintenance.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!