Table of Contents
What “hidden size” and “depth” mean in an MLP
In a multi layer perceptron for tabular or other structured data, depth is the number of hidden layers, and hidden size is the number of units in each hidden layer. If your model is written as a sequence of Linear layers with activations in between, depth counts how many times you apply a hidden Linear plus activation block before the final output layer. Hidden size is the output dimension of each hidden Linear. A common shorthand is to describe an MLP as something like 128, 64, 32, meaning three hidden layers with those widths, followed by an output layer sized to your task.
Start with a simple baseline and grow only when needed
A practical way to choose hidden sizes and depth is to begin with a small, reasonable baseline that you can train quickly, then increase capacity only if you have evidence of underfitting. For structured data, a strong starting point is one to three hidden layers with widths between 64 and 256. Many problems do well with two hidden layers, for example input to 128 to 64 to output, because it is expressive enough to model nonlinear interactions while still being easy to optimize.
If training loss is high and does not improve much even when you train longer, your model may be underpowered, and you can try adding width first, then depth. If training loss becomes very low but validation loss is noticeably worse, you likely have too much capacity for the amount of signal and data you have, and you should reduce width or depth, or apply stronger regularization, which is covered elsewhere.
Increase model capacity only when you see underfitting on the training set. If you already fit the training set well and generalization is poor, adding depth or width usually makes validation performance worse.
Width versus depth, what changes when you adjust each
Increasing width makes each layer able to represent more features in parallel. It often improves performance smoothly and is usually easier to train than making the network much deeper. For beginners, increasing width is the first knob to turn.
Increasing depth increases the number of nonlinear transformations applied in sequence. Depth can represent complex feature hierarchies and interactions more efficiently than a single huge layer, but deeper MLPs can become harder to train and more sensitive to hyperparameters. On structured data, extremely deep plain MLPs are not common unless you add architectural helpers like residual connections, which are beyond a beginner baseline.
A typical progression is to try width scaling before depth scaling, for example moving from 64 to 128 to 256 units per layer, while keeping depth fixed at two layers, before trying three or four layers.
Heuristics that work well for structured data
A simple and commonly effective shape is a “funnel” where layer sizes decrease as you get closer to the output, such as 256 to 128 to 64. This encourages the model to compress learned features into a smaller representation. It also helps keep parameter counts under control.
Another easy pattern is “constant width” for a few layers, such as 128 to 128 to 128. This is convenient to implement and often performs similarly to a funnel at the same scale, especially when you are early in experimentation.
When you have categorical embeddings as inputs, the concatenated embedding vector can be fairly wide. In that case, a first hidden layer of 128 or 256 is often a good starting point, then you can reduce from there.
Do not make the final hidden layer too tiny unless you have a reason. If you compress to something like 4 or 8 units too early, the network can become a bottleneck and underfit even if earlier layers are wide.
Parameter count and memory, why size grows quickly
The number of parameters in a Linear layer from $n_{in}$ to $n_{out}$ is $n_{in} \cdot n_{out}$ for weights plus $n_{out}$ for bias. This means width increases can square your parameter count when you apply them across multiple layers.
For a two hidden layer MLP with input dimension $d$, hidden width $h$, and output dimension $k$, the parameter count is roughly
$$d h + h^2 + h k,$$
ignoring biases. The $h^2$ term is why very wide hidden layers can become expensive quickly. This matters for training speed, GPU memory, and overfitting risk.
Picking sizes based on dataset size and noise level
If you have a small dataset, a smaller network is usually safer. With limited data, a large model can memorize patterns that do not generalize. If you have a larger dataset with many examples and a complex relationship between features and target, you can afford more capacity.
Noise level matters too. When labels are noisy or the target is inherently uncertain, very large models may chase noise. In those cases, moderate widths and shallow depth often generalize better.
A practical tuning recipe for beginners
Choose a baseline such as two hidden layers with 128 and 64 units. Train and evaluate. If both training and validation are poor, increase width, for example 256 and 128, or add one more hidden layer, for example 256, 128, 64. If training improves a lot but validation does not, reduce capacity and rely on regularization strategies covered in other chapters.
When comparing architectures, change only one thing at a time. If you change depth, width, learning rate, and batch size all at once, you will not know what caused any improvement.
When you change depth or width, keep your training setup fixed at first, including optimizer and learning rate. Otherwise, you cannot attribute changes in performance to the architecture.
Implementing flexible depth and width in PyTorch
A clean way to experiment is to build an MLP from a list of hidden sizes. You specify the input dimension, a list like [256, 128, 64], and the output dimension, then create Linear layers accordingly. This keeps architecture search simple and reduces bugs.
In practice, your module often looks like a stack of Linear and activation layers stored in an nn.Sequential, ending with a final Linear to the output. The exact activation and normalization choices are handled in other chapters, but the key point here is that depth and width are easiest to manage when they are data driven by a list of integers rather than hard coded.
When deeper MLPs are not helping
If you find that adding layers does not improve validation performance, it often means the problem does not require that much hierarchical composition, or the data is better served by feature engineering, embeddings, or different model families. For structured data, gradient boosted trees and simpler linear or shallow neural models can be strong baselines, but within this course you should treat that as a reminder to avoid unnecessary complexity.
A good stopping point is when increasing width or depth yields little validation improvement across multiple random seeds, or when training time increases substantially without clear gains.