12.6. Comparing Models Fairly

Why “fair” comparisons matter

A model comparison is fair when the observed difference in performance is most likely caused by the modeling choice you are evaluating, rather than by changes in data, preprocessing, training budget, randomness, or evaluation procedure. In deep learning, small accidental differences can easily create misleading conclusions, especially when runs are noisy. The goal is not to prove one model is always better, but to design an evaluation that isolates the variable you care about and reports results with enough context that someone else could reproduce the comparison.

Fix the evaluation target before training

A fair comparison starts by deciding what “better” means in your project. That usually includes a primary metric, a dataset split policy, and a decision rule for selecting a final checkpoint. If you compare two models but allow each to optimize a different metric, or you pick the best checkpoint using different logic for each, you are not comparing the same target.

Decide whether the primary metric is computed on the validation set for model selection or on the test set for final reporting. Use the test set only once for the final number, after you have finished all decisions.

Keep the data pipeline identical

When comparing architectures or training tricks, keep the data pipeline constant across runs. That includes the exact train, validation, and test splits, preprocessing steps, tokenization or image resizing strategy, normalization statistics, and augmentation policy.

If you use random augmentations, make sure both models see augmentations sampled from the same distribution. You do not need each mini batch to be identical across models, but you should not silently change augmentation strength, crop size, or normalization between experiments unless that is the thing you are testing. If you compute normalization statistics from the training set, compute them once and reuse them for all models.

For class imbalanced problems, use the same sampling strategy and class weights across models. Otherwise you might be changing the effective training objective.

Match the training budget

A very common unfair comparison happens when one model gets more optimization work than another. “Budget” includes the number of epochs, the number of gradient updates, the batch size, and even early stopping rules.

If you compare two models, ensure they train under the same constraints, for example the same maximum number of optimizer steps. If one model trains for 50 epochs and the other for 200 epochs, you are also comparing training time. Sometimes that is valid, but then the conclusion should be stated as a tradeoff between accuracy and compute.

Also match any learning rate schedule behavior that depends on epoch or step counts. If a schedule is defined over 100 epochs and you change the epoch count, you have changed the schedule.

Control randomness and run multiple seeds

Neural network training is stochastic due to initialization, shuffling, dropout, and nondeterministic GPU operations. A single run can be an outlier. For a fair comparison, run each configuration with multiple random seeds and summarize performance with a mean and a measure of variability.

If $m_1, \dots, m_k$ are the metric values across $k$ seeds, report the mean
$$
\bar{m} = \frac{1}{k}\sum_{i=1}^k m_i
$$
and the sample standard deviation
$$
s = \sqrt{\frac{1}{k-1}\sum_{i=1}^k (m_i - \bar{m})^2}.
$$
If you want an approximate 95 percent confidence interval for the mean, you can use
$$
\bar{m} \pm 1.96 \cdot \frac{s}{\sqrt{k}},
$$
which is a common approximation when $k$ is not too small.

Use the same set of seeds for all models so that comparisons are paired as much as possible. Pairing reduces noise because each seed represents a similar “difficulty level” of training randomness.

Choose hyperparameters without giving one model extra help

If you tune hyperparameters, tuning must be done under the same rules for each model. The fairest approach is to give each model the same tuning budget, for example the same number of trials, the same search space size, and the same validation metric.

Avoid tuning one model extensively while leaving the other at default settings, then claiming the tuned one is superior. If you must tune, document the search space and the number of trials.

When budgets are tight, a reasonable approach is to tune a baseline to a solid level and then allow the same limited tuning for each new idea, not unlimited tuning until it wins.

Use consistent checkpoint selection and thresholding

If you select the best epoch based on validation performance, apply the same selection rule to all models, for example “choose the checkpoint with the lowest validation loss” or “highest validation F1.” Do not pick by looking at the test curve.

For classification tasks that require a decision threshold, fix how the threshold is chosen. If you tune the threshold on the validation set for one model, do the same for others. If you use a fixed threshold like 0.5, keep it fixed. Threshold tuning can create large metric swings, so it must be treated as part of the model selection procedure.

Compare on the same metrics and the same slices

Two models can look similar on an overall metric while behaving differently on important subsets, such as rare classes, certain demographic groups, or particular input lengths. For fairness, evaluate both models on the exact same metrics and the same data slices.

If you report macro F1 for one model and micro F1 for another, the comparison is not meaningful. If you report accuracy for one and AUC for the other, you are not measuring the same thing. Decide a primary metric and a small set of secondary metrics, then compute them for all models identically.

Account for compute, memory, and latency when they matter

Sometimes “best” is not only accuracy. If deployment constraints exist, then fairness includes comparing resource costs. Report training time, peak GPU memory, parameter count, and inference latency in a consistent environment.

If you compare inference speed, measure it with the same batch size, precision mode, device, and warmup procedure. Otherwise the numbers are not comparable.

A useful way to frame results is as a Pareto tradeoff, for example accuracy versus latency. A model that is slightly worse in accuracy but much faster might still be preferred, but it should be described as a tradeoff, not as simply better or worse.

Avoid common comparison traps

One trap is reporting the best run out of many for a favored model while reporting an average for the baseline. Another trap is changing multiple things at once, such as architecture plus data augmentation plus learning rate schedule, then attributing the gain to the architecture.

Another trap is failing to check that preprocessing and label handling are identical, for example different tokenization rules, different class to index mappings, or different handling of missing values. These can create large differences that look like model improvements but are actually data pipeline changes.

A practical “fair comparison” checklist for your experiments

Before you trust a model comparison, verify that the dataset splits are identical, the preprocessing and augmentation policies match, the training budget in steps is matched, the optimizer and schedule differences are intentional, the checkpoint selection rule is the same, and results are aggregated across multiple seeds. Then report both performance metrics and, when relevant, compute costs.

When you follow these rules, your comparisons become more reproducible, your conclusions become more reliable, and you waste less time chasing improvements that were actually caused by accidental differences in the experiment setup.

Views: 94