Table of Contents
Why Experiment Tracking Matters
Training a deep learning model is an iterative process where you change one thing, rerun training, and hope the result improves. Without a record of what changed, you quickly end up with results you cannot reproduce, curves you cannot compare, and models you cannot confidently ship. Experiment tracking is the practice of systematically recording what you ran, with what data and settings, what you observed, and which artifacts you produced, so you can answer basic questions later, such as which run produced the best validation score, what hyperparameters were used, and whether a result is real or an accident.
For beginners, the goal is not to build a perfect MLOps system. The goal is to create a simple, consistent habit that prevents confusion and makes progress measurable.
What Counts as an Experiment
An experiment is one training and evaluation attempt under a specific set of conditions. Those conditions include the model architecture choices, training procedure, data preprocessing, and the random state. In practice, you will run many experiments that differ only slightly, such as a different learning rate or a different data augmentation setting. Tracking ensures those small changes are visible and attributable.
An experiment usually contains multiple epochs and yields time series data such as training loss per step, validation loss per epoch, and metrics like accuracy. It also produces artifacts like model weights, plots, and sometimes predictions on a validation set.
If you cannot reconstruct what you did from your records, you did not really track the experiment.
The Core Pieces to Track
At a minimum, you want to capture configuration, environment, results, and artifacts. Configuration includes hyperparameters like learning rate, batch size, number of epochs, optimizer type, weight decay, dropout, and any model size parameters. It also includes data related settings like which dataset version you used, how you split train and validation sets, what transforms were applied, and how text was tokenized if applicable.
Environment refers to the software and hardware context: Python version, PyTorch version, CUDA version if relevant, and the device type. This matters because changes in libraries can change performance and even numerical results.
Results include scalar metrics and learning curves. Scalars are values like final validation accuracy or best validation loss. Curves are the history, for example validation loss over epochs, which often tells you more than a single number.
Artifacts are the tangible outputs you might need later, especially the best model checkpoint, the configuration file used for that run, and optional extras like confusion matrices, example predictions, and plots.
Run Identity and Naming Conventions
Every run needs a unique identity. A simple approach is to create a run directory that includes a timestamp and a short descriptive suffix, such as a model name or dataset tag. What matters is that you can locate a run later and that the identity is stable across your logs and saved files.
A good run name is readable and searchable, but you should not rely on the name alone to store critical information. Names are for humans, configurations are for correctness. When you are in a hurry, you will forget to encode something important in the name.
Do not encode the full experiment in the run name. Always save the full configuration used for training.
Configuration as a First Class Artifact
Treat your configuration as something you can save, load, and compare. Beginners often hardcode values inside notebooks, then later cannot recall which values were used. Instead, keep a single configuration object that is printed at the start of training and saved to disk alongside the model checkpoint.
A configuration should include not only hyperparameters but also derived choices that affect results, such as the input image size after preprocessing, the vocabulary size for a tokenizer, and the label mapping for classification. If a choice can change the model behavior, it belongs in the configuration.
Tracking Randomness and Reproducibility Links
Even when you set seeds, training can vary across runs due to nondeterminism and hardware differences. Experiment tracking does not remove randomness, but it makes it visible. You should record the seed value used for the run and whether you enabled deterministic settings. If you rerun an experiment and get a different result, the record helps you determine whether code, data, or environment changed.
For tracking purposes, the key is to store the seed and the versions. The detailed mechanics of determinism belong elsewhere, but the tracking mindset is that seed and versions are part of the experiment definition.
Metrics, Steps, and Aggregation
When you track metrics, you need to be clear about what the x axis represents. In PyTorch training, you commonly log per batch using a global step counter and per epoch using an epoch index. A global step is typically the number of optimizer updates performed so far. If you use gradient accumulation, the relationship between batches seen and optimizer steps changes, so it is important to log the step definition you use.
Also clarify how metrics are aggregated. For example, an epoch training loss might be the mean of batch losses weighted by batch size. A validation metric should be computed over the full validation set, not averaged per batch in a way that changes with batch sizes. When comparing runs, inconsistent aggregation can produce misleading conclusions.
A metric is only comparable across runs if it is computed the same way on the same data split.
What to Save: Checkpoints and “Best” Models
Tracking experiments is not only about numbers, it is also about preserving the right artifacts. You usually want at least one checkpoint: the model state that achieved the best validation performance according to a chosen criterion. You may also want the final checkpoint at the end of training, which can be useful for debugging divergence or resuming.
The important tracking concept is that you must define what “best” means. For a loss, lower is better. For accuracy, higher is better. For other metrics, you need to specify direction and the exact metric name. Store the best metric value and the epoch it occurred, because two runs can end with similar final numbers but very different best points.
Lightweight Tracking Approaches
You can track experiments with increasing levels of structure. The simplest level is writing to the console and saving a config plus a checkpoint folder. The next level is logging scalars to a file, such as CSV or JSON Lines, where each row includes run id, step, and metric values. This makes it easy to plot later and compare runs in a spreadsheet.
A common next step in PyTorch workflows is using TensorBoard to log scalars, histograms, and images. TensorBoard gives you a practical UI for comparing runs, especially learning curves. The conceptual point is that whichever tool you choose, you should log the same core data: configuration, metrics over time, and artifacts.
Dedicated experiment trackers like MLflow or Weights and Biases add features like a central dashboard, automatic environment capture, and richer comparisons. You do not need them to learn deep learning, but it helps to understand the concept: a run is a structured object with parameters, metrics, and artifacts that can be searched and compared.
Comparing Runs Without Fooling Yourself
Experiment tracking supports fair comparisons. To compare runs, you should keep most factors fixed and vary one or a small number of factors. Record what changed. If you change the dataset split, the preprocessing, and the learning rate all at once, you cannot interpret the result.
It is also important to track the selection process. If you try twenty runs and pick the best one, that best validation score is optimistic by selection. Tracking helps you be honest about how many attempts you made and encourages you to reserve a true test set for a final evaluation, which is discussed elsewhere in the evaluation section.
A Minimal Experiment Record Template
A useful mental template for each run includes the objective, the configuration, the data identity, the results, and the artifacts. The objective is what you were trying to improve, such as validation F1. The data identity is which dataset version and split. The results include best validation metric and the epoch, plus any notes about instability. The artifacts include the best checkpoint and the configuration file.
If you keep these elements consistent, you will be able to answer questions later without rerunning training, and you will build an intuition for which changes reliably help.
Always save, alongside the checkpoint, the exact configuration and the metric used to choose “best.” Without these, a checkpoint is an orphan artifact.