Table of Contents
Purpose of Separate Splits
A model that is evaluated on the same data it learned from can look impressively accurate while still failing on new, unseen examples. The core best practice is to separate your data into splits that serve different roles, training for learning parameters, validation for choosing decisions during development, and test for the final, one time estimate of real world performance. This separation is not just a convention, it is how you prevent information from “leaking” from evaluation into training decisions.
Treat the test set as read only. If you look at test performance and then change anything because of it, your test score is no longer an unbiased estimate.
What Each Split Is For
The training set is used to fit model parameters with gradient based optimization. Anything that directly updates weights should see only training data.
The validation set is used during development to make choices such as model architecture, feature engineering, preprocessing choices, early stopping, learning rate schedules, threshold selection, and hyperparameter tuning. You evaluate repeatedly on validation, so you should expect some “validation overfitting” if you try many alternatives. That is normal, and it is exactly why you need a test set that you do not touch until the end.
The test set is used once, after you have committed to your approach. Its job is to approximate how the system will work on future data from the same distribution.
How Big Should Each Split Be
There is no universal ratio, but the principle is that training needs to be large enough to learn, and validation and test need to be large enough to estimate performance with acceptable noise. Common beginner friendly defaults are 80/10/10 or 70/15/15, but dataset size matters more than the percentage. With very small datasets, your estimates become unstable and cross validation becomes attractive, which is covered elsewhere.
If you are constrained, prioritize keeping a meaningful validation set because you will use it to steer development. For the test set, even a modest sized sample can be useful if it is truly untouched and representative.
Split Before You Fit Anything
A frequent mistake is computing preprocessing statistics using all data and then splitting. Any computation that uses global information, like normalization means and standard deviations, vocabulary building, target encoding, or feature selection, must be fit on the training split only, then applied to validation and test.
In practice, you should split first, then fit preprocessing on training, then transform each split with the fitted preprocessing.
Any operation that “learns from data” must be fit on training only. This includes scalers, imputers, tokenizers built from your corpus, and even decisions like which features to keep.
Stratification and Group Aware Splits
For classification, random splitting can accidentally change class proportions, especially with imbalanced data. Stratified splitting preserves the class distribution across splits, making validation and test metrics more reliable.
Some datasets contain natural groups that must not be split across train and evaluation, such as multiple rows per user, multiple images of the same patient, or repeated measurements over time. If the same group appears in both training and validation, the model can indirectly memorize group specific patterns and your evaluation will be overly optimistic. In these cases, perform group aware splitting so that each group appears in exactly one split.
Time Based Splits for Temporal Data
If your data has a time component and you want to predict future outcomes, random splits can leak future information into training. The best practice is to split by time, training on earlier periods, validating on a later period, and testing on the most recent period. This aligns the evaluation with deployment reality and reveals concept drift issues sooner.
For forecasting and any “predict the future” task, do not shuffle across time when creating train, validation, and test splits.
Avoiding Duplicate and Near Duplicate Leakage
Duplicates can appear in surprising ways, especially in scraped datasets, augmented image sets, or text corpora with repeated templates. If duplicates cross the split boundary, the model can effectively be tested on examples it has already seen. Deduplicate before splitting when possible, or at least ensure duplicates are assigned to the same split. For images and text, also watch for near duplicates, like resized copies or lightly edited versions, which can produce the same leakage effect.
Repeated Peeking and the Final Test
During development you might be tempted to check the test set “just to see.” Each peek increases the chance you will make choices that improve the test score by luck rather than by true generalization. A practical workflow is to use only train and validation until you have a final candidate, optionally run multiple training runs with different random seeds to confirm stability on validation, then evaluate on the test set once.
If your organization requires frequent reporting, you can maintain a public test set for quick checks and a private, final holdout set used only for release decisions. In small personal projects, the simpler rule is usually enough, keep the test set locked away.
Reporting Results Correctly
When you report a final number, it should come from the test set, and the model configuration should be the one selected using validation only. If you do hyperparameter tuning, report that the test metric is computed after selecting hyperparameters on validation.
If you compare many models, it is easy to unintentionally “choose the best on test.” Instead, choose the best on validation, then test only the winner.
Do not use the test set to choose between models. Use validation to choose, and test to confirm.
Practical PyTorch Workflow Tips for Clean Splits
Keep your indices for each split, not just the tensors. Save them to disk so you can reproduce the exact same split later. This is especially important for experiment tracking and for comparing models fairly. If you are using a PyTorch Dataset, create split specific subsets by indexing into a single underlying dataset rather than copying data around, which reduces mistakes.
Also keep the evaluation code path consistent. Use the same preprocessing and the same metric computation for validation and test, and ensure you switch the model into evaluation mode for evaluation so that layers like dropout and batch normalization behave correctly. The details of evaluation mode are covered elsewhere, but the best practice here is consistency, validation and test should reflect the same evaluation conditions.
A Minimal Checklist Before You Trust a Test Score
Confirm that the test set was never used for training, hyperparameter selection, early stopping decisions, or preprocessing fitting. Confirm stratification or group constraints if needed. Confirm time based splitting if applicable. Confirm no duplicates across splits. Confirm you can reproduce the split with saved indices and a fixed seed. Once these are true, your final test result is meaningfully interpretable as an estimate of performance on new data drawn from the same distribution.