8.1. Feature Scaling and Normalization
Table of Contents
Why scaling matters for structured data
Structured or tabular datasets often mix features with very different numeric ranges, such as age in years, income in dollars, counts, and ratios. Gradient based training is sensitive to these scales. If one feature ranges from 0 to 1 and another ranges from 0 to 100,000, the model can spend most of its capacity and gradient updates reacting to the large scale feature, even if it is not the most informative. For multi layer perceptrons, scaling usually makes optimization faster, reduces training instability, and makes learning rate choices less fragile. It also makes regularization behave more predictably, because penalties like weight decay interact with feature magnitudes.
Scaling is also a practical hygiene step: it makes it easier to detect data issues, compare feature importances, and reuse hyperparameters across projects.
The most common normalization choices
A few standard transformations cover most beginner workflows for tabular neural networks.
Standardization, sometimes called z score normalization, transforms a feature $x$ into
$$
x' = \frac{x - \mu}{\sigma}
$$
where $\mu$ is the mean and $\sigma$ is the standard deviation computed on the training set. After standardization, features are roughly centered at zero with unit variance. This is the default choice for many MLPs on continuous numeric features.
Min max scaling maps a feature into a fixed interval, often $[0, 1]$,
$$
x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}
$$
This is useful when you want bounded inputs or you know the feature has a meaningful finite range. It can be sensitive to outliers since $x_{\min}$ and $x_{\max}$ can be extreme.
Robust scaling uses the median and interquartile range to reduce the effect of outliers,
$$
x' = \frac{x - \text{median}(x)}{\text{IQR}(x)}
$$
This is a good option when numeric features have heavy tails or occasional extreme values.
Log type transforms are often applied before standardization for strictly positive skewed features such as counts or prices, for example $x' = \log(1 + x)$. The goal is to compress large values and make the distribution less skewed.
:::danger :::
Compute normalization statistics using only the training split, then reuse the same statistics to transform validation and test data. Using validation or test data to fit $\mu$, $\sigma$, $x_{\min}$, or $x_{\max}$ leaks information and makes evaluation overly optimistic.
What to scale and what not to scale
Continuous numeric features are the main target for scaling.
Binary indicator features that are already 0 or 1 typically do not need scaling. One hot encoded categorical features are also already on a comparable scale, so scaling is usually unnecessary and can even blur their interpretation.
Categorical features that you plan to feed through embedding layers should not be standardized as if they were continuous. They should be represented as integer indices and handled by embeddings, which is covered in the embeddings chapter.
Targets can also be scaled in regression. Standardizing the target $y$ can make training easier when $y$ spans large ranges. If you do this, you must unscale predictions back to the original units for reporting and downstream use.
:::danger :::
Never standardize categorical IDs that are meant to be indices. Converting category IDs to standardized floats destroys the discrete meaning and will break embeddings or make the model learn nonsense numeric relationships between category labels.
Implementing scaling correctly in a PyTorch workflow
PyTorch does not require you to use any particular preprocessing library. A common pattern is to compute statistics once from the training data, store them, and apply the transform inside the dataset, a transform, or just before you create tensors.
If $X_{\text{train}}$ is a training tensor of shape $(N, D)$ with float dtype, per feature means and standard deviations are computed along the batch dimension:
$\mu = \text{mean}(X_{\text{train}}, \text{dim}=0)$ and $\sigma = \text{std}(X_{\text{train}}, \text{dim}=0)$. Then each batch $X$ is transformed as $(X - \mu) / \sigma$ using broadcasting.
A small numerical detail matters in practice: if a feature has near zero variance, $\sigma$ can be zero or extremely small, which will produce infinities or very large values. The standard fix is to clamp or add a small epsilon:
$$
x' = \frac{x - \mu}{\sigma + \varepsilon}
$$
where $\varepsilon$ might be $10^{-6}$ or $10^{-8}$.
:::danger :::
Always guard against zero variance features by using $\sigma + \varepsilon$ or by removing constant columns. Otherwise you can create NaNs or infs that silently ruin training.
For validation and test, you apply the same $\mu$ and $\sigma$ computed from training. Store them alongside your model or in the training checkpoint so inference uses identical preprocessing.
Where normalization lives in the pipeline
You can normalize at different points, and the best choice depends on your project constraints.
If your dataset fits in memory and you are doing classical tabular modeling, you can normalize the full arrays before building the PyTorch dataset. This is simple and fast.
If you want to keep raw data unchanged, normalize inside the Dataset class so every item is transformed on the fly. This makes it harder to accidentally mismatch preprocessing during inference, because the dataset owns the logic.
If you deploy a model, you must ensure the same preprocessing runs in production. That often means saving $\mu$ and $\sigma$ and implementing the transform in your inference code. If you export the model, you still need to handle preprocessing unless you explicitly bake it into the model forward pass.
A quick checklist for beginners
Scaling is not about making the data look nicer, it is about making optimization behave well. Choose a scaling method, fit it on the training split only, apply it consistently everywhere, and protect against zero variance features. If training is unstable, loss is not decreasing, or learning rate seems extremely sensitive, checking and fixing feature scaling is one of the highest return steps for structured data models.
Views: 0
Where to Move
Move chapter:
- ☰ 1. Orientation and Learning Setup
- ☰ 1.1. Course Goals and How to Use This Outline
- ☰ 1.2. What You Need to Know Before Starting
- ☰ 1.3. Installing Python and PyTorch
- ☰ 1.4. Choosing CPU vs CUDA vs Apple Silicon
- ☰ 1.5. Using Jupyter, VS Code, and Scripts
- ☰ 1.6. Reproducibility, Seeds, and Determinism
-
- ☰ 2. Python and PyTorch Fundamentals
- ☰ 2.1. Tensors and Basic Operations
- ☰ 2.2. Tensor Shapes, Dtypes, and Devices
- ☰ 2.3. Indexing, Slicing, and Broadcasting
- ☰ 2.4. Autograd and Computational Graphs
- ☰ 2.5. Common Tensor Pitfalls and Debugging
- ☰ 2.6. Saving and Loading Tensors
-
- ☰ 3. Neural Network Building Blocks
- ☰ 3.1. Linear Layers and Affine Transforms
- ☰ 3.2. Activation Functions
- ☰ 3.3. Loss Functions
- ☰ 3.4. Optimizers and Learning Rates
- ☰ 3.5. Regularization Basics
- ☰ 3.6. Weight Initialization Concepts
-
- ☰ 4. First Models End to End
- ☰ 4.1. A Simple Regression Model
- ☰ 4.2. A Simple Classification Model
- ☰ 4.3. Data Splitting and Validation Sets
- ☰ 4.4. Training Loop Structure
- ☰ 4.5. Tracking Loss and Metrics
- ☰ 4.6. Basic Overfitting and Underfitting Checks
-
- ☰ 5. Data Handling with PyTorch
- ☰ 5.1. Datasets and DataLoaders
- ☰ 5.2. Transforms and Preprocessing
- ☰ 5.3. Collate Functions and Batching
- ☰ 5.4. Shuffling, Sampling, and Class Imbalance
- ☰ 5.5. Working with Tabular, Text, and Image Data
- ☰ 5.6. Data Pipeline Performance Basics
-
- ☰ 6. Training Loops in Practice
- ☰ 6.1. Forward Pass and Backward Pass
- ☰ 6.2. Gradient Zeroing and Accumulation
- ☰ 6.3. Device Placement and Mixed Precision Basics
- ☰ 6.4. Learning Rate Schedules
- ☰ 6.5. Gradient Clipping
- ☰ 6.6. Logging with TensorBoard
-
- ☰ 7. Debugging and Model Troubleshooting
- ☰ 7.1. Verifying Shapes and NaNs
- ☰ 7.2. Checking Gradients and Vanishing Gradients
- ☰ 7.3. Sanity Checks and Overfit a Small Batch
- ☰ 7.4. Common Causes of Training Instability
- ☰ 7.5. Interpreting Learning Curves
- ☰ 7.6. Practical Tips for Faster Iteration
-
- ☰ 8. Feedforward Networks for Structured Data
- ☰ 8.1. Feature Scaling and Normalization
- ☰ 8.2. Embeddings for Categorical Features
- ☰ 8.3. Multi-Layer Perceptrons in PyTorch
- ☰ 8.4. Choosing Hidden Sizes and Depth
- ☰ 8.5. Evaluation Metrics for Classification and Regression
- ☰ 8.6. Model Export and Inference Basics
-
- ☰ 9. Convolutional Neural Networks for Images
- ☰ 9.1. Convolutions, Padding, and Stride
- ☰ 9.2. Pooling and Feature Maps
- ☰ 9.3. Building a Basic CNN in PyTorch
- ☰ 9.4. Data Augmentation for Vision
- ☰ 9.5. Transfer Learning with Pretrained Models
- ☰ 9.6. Fine-Tuning and Freezing Layers
-
- ☰ 10. Recurrent and Sequence Models
- ☰ 10.1. Sequences, Tokens, and Padding
- ☰ 10.2. RNNs, LSTMs, and GRUs
- ☰ 10.3. Handling Variable-Length Sequences
- ☰ 10.4. Teacher Forcing and Sequence Losses
- ☰ 10.5. Basic Text Classification Pipeline
- ☰ 10.6. Simple Sequence-to-Sequence Overview
-
- ☰ 11. Attention and Transformers Fundamentals
- ☰ 11.1. Why Attention Helps
- ☰ 11.2. Self-Attention Intuition
- ☰ 11.3. Transformer Building Blocks
- ☰ 11.4. Using Transformer Models in PyTorch
- ☰ 11.5. Fine-Tuning for Text Classification
- ☰ 11.6. Managing Tokenizers and Batching
-
- ☰ 12. Model Evaluation and Experiment Management
- ☰ 12.1. Train, Validation, and Test Best Practices
- ☰ 12.2. Cross-Validation Basics
- ☰ 12.3. Calibration and Thresholding
- ☰ 12.4. Error Analysis and Confusion Matrices
- ☰ 12.5. Experiment Tracking Concepts
- ☰ 12.6. Comparing Models Fairly
-
- ☰ 13. Saving, Loading, and Deployment Readiness
- ☰ 13.1. Saving Model Weights and Full Checkpoints
- ☰ 13.2. Loading for Inference and Resuming Training
- ☰ 13.3. TorchScript and Tracing Basics
- ☰ 13.4. Exporting to ONNX Overview
- ☰ 13.5. Batch Inference and Latency Considerations
- ☰ 13.6. Reproducible Packaging of Code and Weights
-
- ☰ 14. Performance and Scaling Basics
- ☰ 14.1. Profiling and Bottleneck Identification
- ☰ 14.2. DataLoader Optimization
- ☰ 14.3. GPU Utilization and Memory Management
- ☰ 14.4. Gradient Accumulation for Large Batches
- ☰ 14.5. Distributed Training Concepts Overview
- ☰ 14.6. Practical Guidelines for Faster Training
-
- ☰ 15. Best Practices and Next Steps
- ☰ 15.1. Choosing Hyperparameters Systematically
- ☰ 15.2. Regularization and Generalization Toolkit
- ☰ 15.3. Responsible Use and Dataset Considerations
- ☰ 15.4. Reading PyTorch Documentation Effectively
- ☰ 15.5. Suggested Projects for Beginners
- ☰ 15.6. Pathways to Advanced Topics
-
Comments
Please login to add a comment.
Don't have an account? Register now!