Table of Contents
Why scaling matters for structured data
Structured or tabular datasets often mix features with very different numeric ranges, such as age in years, income in dollars, counts, and ratios. Gradient based training is sensitive to these scales. If one feature ranges from 0 to 1 and another ranges from 0 to 100,000, the model can spend most of its capacity and gradient updates reacting to the large scale feature, even if it is not the most informative. For multi layer perceptrons, scaling usually makes optimization faster, reduces training instability, and makes learning rate choices less fragile. It also makes regularization behave more predictably, because penalties like weight decay interact with feature magnitudes.
Scaling is also a practical hygiene step: it makes it easier to detect data issues, compare feature importances, and reuse hyperparameters across projects.
The most common normalization choices
A few standard transformations cover most beginner workflows for tabular neural networks.
Standardization, sometimes called z score normalization, transforms a feature $x$ into
$$
x' = \frac{x - \mu}{\sigma}
$$
where $\mu$ is the mean and $\sigma$ is the standard deviation computed on the training set. After standardization, features are roughly centered at zero with unit variance. This is the default choice for many MLPs on continuous numeric features.
Min max scaling maps a feature into a fixed interval, often $[0, 1]$,
$$
x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}
$$
This is useful when you want bounded inputs or you know the feature has a meaningful finite range. It can be sensitive to outliers since $x_{\min}$ and $x_{\max}$ can be extreme.
Robust scaling uses the median and interquartile range to reduce the effect of outliers,
$$
x' = \frac{x - \text{median}(x)}{\text{IQR}(x)}
$$
This is a good option when numeric features have heavy tails or occasional extreme values.
Log type transforms are often applied before standardization for strictly positive skewed features such as counts or prices, for example $x' = \log(1 + x)$. The goal is to compress large values and make the distribution less skewed.
:::danger :::
Compute normalization statistics using only the training split, then reuse the same statistics to transform validation and test data. Using validation or test data to fit $\mu$, $\sigma$, $x_{\min}$, or $x_{\max}$ leaks information and makes evaluation overly optimistic.
What to scale and what not to scale
Continuous numeric features are the main target for scaling.
Binary indicator features that are already 0 or 1 typically do not need scaling. One hot encoded categorical features are also already on a comparable scale, so scaling is usually unnecessary and can even blur their interpretation.
Categorical features that you plan to feed through embedding layers should not be standardized as if they were continuous. They should be represented as integer indices and handled by embeddings, which is covered in the embeddings chapter.
Targets can also be scaled in regression. Standardizing the target $y$ can make training easier when $y$ spans large ranges. If you do this, you must unscale predictions back to the original units for reporting and downstream use.
:::danger :::
Never standardize categorical IDs that are meant to be indices. Converting category IDs to standardized floats destroys the discrete meaning and will break embeddings or make the model learn nonsense numeric relationships between category labels.
Implementing scaling correctly in a PyTorch workflow
PyTorch does not require you to use any particular preprocessing library. A common pattern is to compute statistics once from the training data, store them, and apply the transform inside the dataset, a transform, or just before you create tensors.
If $X_{\text{train}}$ is a training tensor of shape $(N, D)$ with float dtype, per feature means and standard deviations are computed along the batch dimension:
$\mu = \text{mean}(X_{\text{train}}, \text{dim}=0)$ and $\sigma = \text{std}(X_{\text{train}}, \text{dim}=0)$. Then each batch $X$ is transformed as $(X - \mu) / \sigma$ using broadcasting.
A small numerical detail matters in practice: if a feature has near zero variance, $\sigma$ can be zero or extremely small, which will produce infinities or very large values. The standard fix is to clamp or add a small epsilon:
$$
x' = \frac{x - \mu}{\sigma + \varepsilon}
$$
where $\varepsilon$ might be $10^{-6}$ or $10^{-8}$.
:::danger :::
Always guard against zero variance features by using $\sigma + \varepsilon$ or by removing constant columns. Otherwise you can create NaNs or infs that silently ruin training.
For validation and test, you apply the same $\mu$ and $\sigma$ computed from training. Store them alongside your model or in the training checkpoint so inference uses identical preprocessing.
Where normalization lives in the pipeline
You can normalize at different points, and the best choice depends on your project constraints.
If your dataset fits in memory and you are doing classical tabular modeling, you can normalize the full arrays before building the PyTorch dataset. This is simple and fast.
If you want to keep raw data unchanged, normalize inside the Dataset class so every item is transformed on the fly. This makes it harder to accidentally mismatch preprocessing during inference, because the dataset owns the logic.
If you deploy a model, you must ensure the same preprocessing runs in production. That often means saving $\mu$ and $\sigma$ and implementing the transform in your inference code. If you export the model, you still need to handle preprocessing unless you explicitly bake it into the model forward pass.
A quick checklist for beginners
Scaling is not about making the data look nicer, it is about making optimization behave well. Choose a scaling method, fit it on the training split only, apply it consistently everywhere, and protect against zero variance features. If training is unstable, loss is not decreasing, or learning rate seems extremely sensitive, checking and fixing feature scaling is one of the highest return steps for structured data models.