Table of Contents
What “Transforms” Mean in a PyTorch Data Pipeline
Transforms are the part of your data pipeline that convert raw samples into the tensors your model expects. A raw sample might be an image file on disk, a row from a CSV, or a piece of text. Preprocessing is the set of operations that make those samples consistent, numeric, and easier for a neural network to learn from.
A transform is typically applied per sample, inside your Dataset.__getitem__, before batching happens. This keeps your DataLoader simple and makes it easy to reuse the same dataset with different preprocessing choices for training and validation.
Core Goals of Preprocessing
Preprocessing usually has three goals. The first goal is converting data into tensors with the right dtype and shape. The second goal is scaling inputs to a numeric range that makes optimization stable. The third goal is applying training only randomness, such as augmentation, to improve generalization.
Important rule: keep preprocessing deterministic for validation and test sets. Random augmentations belong in the training pipeline only.
Common Transform Patterns for Images
For vision tasks, transforms often include loading an image, converting it to a tensor, resizing or cropping, scaling pixel values, and normalizing with mean and standard deviation.
A typical modern pattern uses torchvision.transforms and applies a composed transform to each sample. The exact choice depends on your model and dataset, but the structure usually looks like this in a dataset:
from torchvision import transforms
train_tfms = transforms.Compose([
transforms.Resize((224, 224)),
transforms.RandomHorizontalFlip(p=0.5),
transforms.ToTensor(),
transforms.Normalize(mean=(0.485, 0.456, 0.406),
std=(0.229, 0.224, 0.225)),
])
val_tfms = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=(0.485, 0.456, 0.406),
std=(0.229, 0.224, 0.225)),
])
Normalization is especially important when you use pretrained CNN backbones because they typically assume a specific input distribution. ToTensor() also changes the shape convention to channels first, so an image becomes C x H x W, and scales pixel values to [0, 1] when the input is a PIL image.
Important rule: if you normalize inputs during training, you must apply the exact same normalization during validation, testing, and inference.
Feature Scaling for Tabular Data
Tabular inputs often need scaling because different columns may have very different ranges. A neural network can struggle when one feature is in the thousands and another is between zero and one.
A standard approach is to compute scaling statistics on the training split only, then reuse them everywhere else. For a numerical feature $x$, standardization is:
$$x' = \frac{x - \mu}{\sigma}$$
where $\mu$ and $\sigma$ are the mean and standard deviation computed from the training data. Another common option is min max scaling:
$$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$
In PyTorch pipelines, scaling can happen inside the dataset, but the statistics should be prepared outside, then injected into the dataset so that the dataset stays consistent.
Important rule: never compute scaling statistics on validation or test data. That leaks information and makes results look better than they are.
Categorical Preprocessing and Label Encoding
Categorical features need conversion to integers before they can be used. A common preprocessing step is to map each category string to an integer id. If your model uses embeddings later, the output of preprocessing is usually a torch.long tensor of ids.
For targets in classification, labels are also commonly encoded as integers 0..K-1. Ensure the dtype matches the loss you plan to use. For example, many classification losses expect class indices as torch.long, not one hot vectors.
Text Preprocessing at a High Level
Text preprocessing often means tokenization, mapping tokens to ids, and handling unknown tokens. In many workflows this is done with a tokenizer object, producing a list of integer ids per sample. Padding to equal lengths is usually handled at batching time, not inside a single sample transform, because different batches may have different maximum lengths.
This chapter focuses on where preprocessing fits and how to structure it. The details of padding and batching behavior belong with batching and collate functions.
Writing Your Own Transform Functions
A transform can be any callable that takes one sample and returns a transformed sample. This is useful for custom logic like clipping values, replacing missing values, applying a log transform, or computing derived features.
A simple example for tabular tensors might look like this:
class Standardize:
def __init__(self, mean, std, eps=1e-8):
self.mean = mean
self.std = std
self.eps = eps
def __call__(self, x):
return (x - self.mean) / (self.std + self.eps)
This pattern keeps preprocessing reusable and testable. You can also compose multiple custom transforms by calling them in sequence, similar to Compose.
Important rule: your transform should not change the meaning of the target unintentionally. If you augment inputs, keep labels consistent with the augmentation.
Where Transforms Live in a Dataset
In practice, transforms are stored as attributes on your dataset, and applied inside __getitem__. Conceptually, the dataset does: read raw sample, apply transform to inputs, apply any target transform if needed, then return tensors.
A common structure is:
class MyDataset(torch.utils.data.Dataset):
def __init__(self, items, transform=None, target_transform=None):
self.items = items
self.transform = transform
self.target_transform = target_transform
def __getitem__(self, idx):
x, y = self.items[idx]
if self.transform is not None:
x = self.transform(x)
if self.target_transform is not None:
y = self.target_transform(y)
return x, y
def __len__(self):
return len(self.items)
This separation helps you reuse the same raw data with different training and validation behavior by swapping transform.
Determinism, Randomness, and Reproducibility in Transforms
Many augmentations are random, such as random crops or flips. If you need reproducible runs, you should control seeds, and be aware that multiple data loader workers introduce additional randomness sources. The key practical point for this chapter is to keep your training transforms potentially stochastic and your evaluation transforms deterministic, and to avoid any preprocessing that depends on the current batch.
Important rule: do not make preprocessing depend on the current batch contents unless you are intentionally implementing a batch dependent method. Most preprocessing should be per sample and consistent across batches.
A Practical Checklist for Beginners
When your model training behaves strangely, preprocessing is a frequent culprit. Check that tensor shapes match what the model expects, dtypes are correct, and normalization is consistent across splits. Verify that you are not accidentally applying training augmentation to validation data, and that any statistics used for scaling were computed from training data only.
If you keep transforms simple, split aware, and consistent, the rest of your training pipeline becomes much easier to debug and trust.