Table of Contents
Tabular Data Pipelines
Tabular data usually arrives as rows where each row is one example and columns are features. In PyTorch, the goal is to turn each row into tensors that a model can consume, typically a single feature tensor and a target tensor. A common pattern is to separate numeric features from categorical features, because numeric values are typically fed as floating point tensors, while categorical values are usually represented as integer indices so they can be embedded later.
When you implement a tabular dataset, you typically store your data in memory as arrays, tensors, or a DataFrame, then in __getitem__ you convert a row into torch.Tensor objects with consistent dtypes. Numeric features are usually torch.float32. Targets depend on the task, for example regression targets are commonly torch.float32, binary classification targets might be torch.float32 with a sigmoid based loss, and multiclass classification targets are commonly integer class indices with torch.long.
For multiclass classification with CrossEntropyLoss, targets must be class indices with dtype torch.long, not one hot vectors.
A tabular sample often returns more than one tensor, for example (x_num, x_cat, y). The default collate behavior of a DataLoader will batch each component separately, producing tensors shaped like [batch_size, num_numeric_features] and [batch_size, num_categorical_features]. This is a clean way to keep numeric and categorical data distinct without writing a custom collate function.
Missing values require a deliberate strategy before you ever reach the model. For numeric features, common beginner friendly choices are filling with a constant, such as zero, or with a precomputed statistic, such as the training set mean. For categorical features, it is common to map missing to a dedicated category like "__UNK__" and assign it its own index.
Text Data Pipelines
Text models need tokenization, which converts strings into sequences of token ids. Regardless of whether you use word level tokens, subword tokens, or character tokens, you will end up with variable length sequences, and variable length sequences are the main practical difference from tabular data.
A typical Dataset for text returns a token id tensor and a label. Token ids are almost always torch.long because they are used to index an embedding table. Labels follow the same dtype rules as in tabular classification.
Token id tensors for embeddings must be integer type, typically torch.long. Passing floating point token ids will error or produce incorrect behavior.
Because sequences have different lengths, batching requires padding. The simplest approach is to pad each batch to the length of the longest sequence in that batch. You can do this by writing a custom collate_fn that takes a list of samples, finds the maximum length, pads shorter sequences with a pad_id, then stacks them into a single tensor of shape [batch_size, max_len]. Many training setups also create an attention mask or padding mask tensor of shape [batch_size, max_len] that is 1 where tokens are real and 0 where tokens are padding, so the model can ignore padded positions.
Always ensure padding uses a dedicated pad_id and that your model or loss ignores padded positions. If you compute a sequence loss over padded tokens, training can fail silently.
If you are building a language modeling or sequence labeling pipeline, your dataset may return both inputs and targets as token sequences. In those cases, it is common to align targets by shifting, for example predicting the next token. The important practical point is to keep shapes consistent across the batch after padding and to verify that loss is only computed on non padded positions.
Image Data Pipelines
Image data typically starts as files on disk. A standard pipeline loads an image, converts it to a tensor, applies preprocessing, and returns the tensor plus a label. Image tensors in PyTorch are usually shaped [C, H, W], where C is the number of channels. When batched by a DataLoader, the tensor becomes [batch_size, C, H, W].
The most common beginner friendly approach is to use torchvision datasets or a custom dataset that reads file paths and labels, then uses torchvision.transforms to apply preprocessing. A crucial detail is ensuring consistent size across a batch. That usually means resizing or cropping images to a fixed height and width so the default collate function can stack them.
A batch requires all image tensors to have the same shape. If images have different sizes and you do not resize or crop, DataLoader will fail when stacking.
Image pixel value ranges also matter. Many transforms convert images to float32 tensors in the range $[0, 1]`. Some pretrained models expect normalization with a specific mean and standard deviation per channel, so the preprocessing must match what the model expects. Even without pretrained models, consistent scaling is important so that gradients behave sensibly.
Labels for images follow the same conventions as tabular. For multiclass classification, labels are typically integer class indices of dtype torch.long. For multi label classification, labels are often a float tensor of zeros and ones.
Choosing What Each Sample Returns
Across tabular, text, and image datasets, the key design decision is the structure of one sample. A good rule is that __getitem__ should return exactly what the training step needs, without extra work inside the training loop. For tabular, that might be (x, y) or (x_num, x_cat, y). For text, it might be (input_ids, attention_mask, y) if you need a mask. For images, it is often (image_tensor, y).
Consistency across samples is essential. Every field returned by __getitem__ should have consistent dtype and rank, and should be easy for the DataLoader to stack or to pad through a collate function. When in doubt, print the shapes and dtypes of a single sample and of one batch before training.
If a model trains poorly or errors, first verify a single batch: tensor shapes, dtypes, value ranges, and that labels match the loss expectations.
A Minimal Mental Checklist per Modality
For tabular data, ensure numeric features are float32, categorical features are integer indices, and targets match the chosen loss. For text, ensure token ids are long, implement padding at batch time, and make sure the model or loss ignores padding. For images, ensure tensors are [C, H, W], sizes are consistent via resize or crop, and preprocessing matches the model expectations.