15.1 Choosing Hyperparameters Systematically

What Hyperparameters Are and Why a System Helps

Hyperparameters are choices you make before and around training that shape how learning happens. They include the learning rate, batch size, model width and depth, optimizer settings, dropout rate, weight decay, and training duration. Unlike model parameters, they are not learned by gradient descent, but they strongly affect whether training converges, how fast it converges, and how well the model generalizes.

A systematic approach matters because most hyperparameters interact. Changing batch size can change the effective noise in gradients, which often changes the best learning rate. Adding weight decay can change the best learning rate schedule. Increasing model size can require different regularization and training time. The goal is to reduce wasted trials and to make decisions you can justify and reproduce.

Establish a Strong Baseline First

Start with a baseline configuration that is intentionally simple and stable. Keep the model modest, use an optimizer with sensible defaults, and train with a standard schedule. The purpose is not to win, it is to create a reference point you can compare against fairly.

A practical baseline usually includes a fixed data split, a fixed evaluation metric, fixed preprocessing, and a training loop that logs the same quantities each run. If your baseline is unstable, hyperparameter search will be noisy and misleading.

Define the Objective and the Budget

Hyperparameter tuning needs a target and a constraint. The target is what you will optimize, typically validation loss or a validation metric such as accuracy, F1, or RMSE. The constraint is your budget, which can be time, number of trials, or total GPU hours. Your budget determines whether you can afford long full training runs or whether you need early stopping and shorter proxy runs.

When your budget is small, you should prioritize hyperparameters with the largest impact per unit effort, usually learning rate, batch size, and training length, before fine-grained regularization tweaks.

Use a Two-Phase Tuning Strategy

A reliable strategy is to tune in two phases. In the exploration phase, you try broad ranges with cheap runs to identify promising regions. In the refinement phase, you narrow the ranges and run longer, closer-to-final training.

In exploration, prefer short runs that are long enough to reveal whether optimization is working. You are not trying to find the best final accuracy yet, you are trying to find settings that learn at all and do not diverge. In refinement, increase training epochs, evaluate more carefully, and reduce randomness by using repeated runs for top candidates if you can afford it.

Prioritize High-Impact Hyperparameters

Learning rate is usually the most important. If training is slow, unstable, or stuck, learning rate is the first knob. Batch size is next, because it changes optimization dynamics and throughput. Training duration and learning rate schedule control how far you let optimization proceed and how you reduce step sizes over time. Regularization settings like weight decay and dropout typically come after you have a learning rate that trains cleanly.

Model capacity, such as number of layers and hidden units, should be tuned with care. Bigger models often look better early but can overfit or require stronger regularization and longer training. If you change capacity, expect to retune learning rate and regularization.

Choose Search Spaces That Make Sense

Many hyperparameters work best on a multiplicative scale, so you should search them on a log scale rather than linearly. This is especially true for learning rate and weight decay.

A common pattern is to sample learning rate as $10^u$ where $u$ is uniform over an interval like $[-5, -1]$, which corresponds to learning rates between $10^{-5}$ and $10^{-1}$. Weight decay often benefits from a similar log search. For dropout, a linear range is usually fine because it naturally lives between 0 and 1.

Prefer Random Search Over Small Grid Search

For beginners, it is tempting to use a small grid, but grids spend too many trials on unimportant combinations and too few on critical dimensions like learning rate. Random search is often more efficient because it explores more unique values in the dimensions that matter.

If you run $N$ trials, random search gives you $N$ distinct learning rates. A grid of size $k \times k$ gives you only $k$ learning rates, even though learning rate may be the dominant factor.

Use Early Stopping and Pruning Carefully

Early stopping can save time by halting runs that clearly underperform. It is most useful in exploration. However, aggressive early stopping can eliminate configurations that learn slowly at first but end up better. A safer version is to allow a minimum number of epochs before judging.

If you implement pruning, base it on a smoothed validation metric rather than a single noisy point, and compare against a reasonable baseline rather than the best run so far, which can be an outlier.

Control Randomness So You Can Compare Fairly

Hyperparameter comparisons are only meaningful if the runs are comparable. Use the same train, validation split and the same preprocessing. Keep the evaluation cadence the same. Fix seeds when you are comparing close contenders to reduce variance, then later test robustness by changing seeds.

If a metric fluctuates heavily across runs, consider reporting the mean over multiple seeds for top configurations. This is more expensive, so reserve it for the final candidates.

Track Every Trial Like an Experiment

Each trial should record the hyperparameters, the code version, the data version, the seed, and the resulting metrics. Without this, you cannot learn from your search and you cannot reproduce the result.

At minimum, save a small structured record per run, such as a JSON file, and log learning curves. When a configuration is good, also save the checkpoint and the exact hyperparameters used. The point is to be able to answer, later, exactly what you did.

A Practical Tuning Order for Beginners

A simple, systematic order is to first find a learning rate that produces smooth, decreasing training loss without instability. Next choose a batch size that fits memory and gives good throughput, then retune learning rate if needed. Then decide a training duration and a schedule so validation performance stabilizes. Only then tune regularization such as weight decay and dropout to improve validation performance without hurting optimization. Finally, tune model size to match the complexity of the task, retuning the earlier knobs as necessary.

This order works because it separates optimization problems from generalization problems. If optimization is failing, regularization tuning will not fix it. If optimization is healthy, regularization can help validation performance.

Know When to Stop Tuning

Stop when improvements are smaller than your noise level or smaller than what matters for your application. If changing seeds moves your metric by 0.3 points, an apparent improvement of 0.1 is not meaningful. Also stop when the cost of additional search is higher than the value of a small improvement.

When you settle on final hyperparameters, run a final training with the chosen configuration and evaluate once on the test set following your evaluation protocol.