Table of Contents
The Core Problem: Different Lengths, Same Batch
Sequence models almost always want batched tensors for speed. Batching assumes every example has the same shape, but real sequences rarely do. One sentence might have 5 tokens, another 30. One audio clip might be 1.2 seconds, another 4.0 seconds. The practical solution is to pad shorter sequences up to a common length, then make sure the model and the loss function do not treat padding as real content.
In PyTorch, “handling variable length” usually means combining three ingredients: padding to create a rectangular batch tensor, keeping the true lengths, and using either packing utilities or masking so computations ignore padded positions.
Padding and Length Tensors
Assume you have tokenized sequences like [[4, 8, 2], [7, 1, 9, 3, 5]]. To batch them, you pad the shorter one to match the longest length in the batch, producing something like [[4, 8, 2, 0, 0], [7, 1, 9, 3, 5]], where 0 is a chosen padding token id. Alongside this padded tensor, you keep a length tensor, here [3, 5], representing the number of real tokens per sequence.
Padding is not optional for batching, but padding must be ignored by the model or the loss. If you let the network learn from padding, training becomes noisy and metrics become misleading.
For text, padding typically uses a dedicated token id such as pad_id. For continuous signals, padding is often zeros, but you still need a mask or lengths to prevent the model from treating zeros as meaningful.
Packing for RNNs: `pack_padded_sequence` and `pad_packed_sequence`
Classic PyTorch RNN modules like nn.RNN, nn.GRU, and nn.LSTM support packed sequences, which are a compact representation that skips padded timesteps during computation. This is often the cleanest approach when using RNN family layers.
The typical flow is: embed or project your padded batch to a feature tensor, pack it using the true lengths, run the RNN on the packed object, then optionally unpack back to padded form.
Conceptually, if your padded embedded batch is x with shape (batch, max_len, d) when batch_first=True, and lengths is shape (batch,), then you do:
Use torch.nn.utils.rnn.pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False) to create a PackedSequence.
Feed it into your RNN, for example packed_out, (h, c) = lstm(packed_x).
If you need per timestep outputs aligned to the padded shape, unpack via pad_packed_sequence(packed_out, batch_first=True).
lengths must be on the CPU for pack_padded_sequence in many setups. A common failure mode is keeping lengths on the GPU and getting a runtime error.
The enforce_sorted=False option allows you to skip sorting sequences by length. Internally PyTorch can handle it, but if you set enforce_sorted=True then you must sort the batch by descending length and remember to unsort outputs back to the original order.
Which Output Do You Actually Need?
Variable length handling is easier if you are clear about what the task needs.
If you are doing sequence classification, you often want one vector per sequence. With LSTMs, a standard choice is the final hidden state per sequence. With packing, PyTorch ensures the “final” state corresponds to the last real token, not padding. That is exactly what you want.
If you are doing token level prediction, you need logits for each timestep. In that case, you usually unpack the packed outputs back into a padded tensor of shape (batch, max_len, hidden) and then apply a linear layer to get (batch, max_len, vocab_or_classes). You then need masking in the loss so padded positions do not contribute.
Masking: The Universal Tool
Even if you use packing, you still often need masking when computing losses or metrics. If you do not use packing, masking is essential.
A mask is a boolean tensor of shape (batch, max_len) where True marks real tokens and False marks padding. You can build it from lengths by comparing a range tensor to each length.
Once you have a mask, you can zero out contributions from padded positions, or you can select only valid positions for loss computation.
When reporting accuracy, F1, or token level metrics, you must exclude padded positions. Otherwise your model can appear to perform well by “predicting padding” correctly.
Loss Computation for Variable Length Sequences
For token level classification, you typically have logits of shape (batch, max_len, C) and targets of shape (batch, max_len). Many PyTorch losses like nn.CrossEntropyLoss expect inputs shaped (N, C) and targets (N,). A common pattern is to flatten batch and time into a single dimension, then ignore padded targets.
If your padding target id is pad_id, you can use nn.CrossEntropyLoss(ignore_index=pad_id). Then you reshape logits to (batch max_len, C) and targets to (batch max_len,). Positions with target pad_id do not contribute to the loss.
For token losses, set ignore_index to the padding label id. This is the simplest and most reliable way to prevent padding from affecting gradients.
For regression at each timestep, there is no ignore_index equivalent. In that case you apply a mask manually, for example computing an elementwise loss and then averaging only over valid positions. The important idea is that the denominator should be the number of valid tokens, not batch * max_len.
Attention to Shapes and `batch_first`
RNN modules can be configured with batch_first=True, which makes inputs and outputs (batch, time, features). If batch_first=False, shapes become (time, batch, features). Packing and unpacking must use the same convention.
Mixing conventions is a common source of silent shape bugs. Decide early, then stick with it consistently across your dataset, collate function, and model.
If you pack sequences, the batch_first flag must match the actual layout of your tensor. A mismatch can produce incorrect results without an obvious error.
Sorting by Length: When and Why
If you do not use enforce_sorted=False, you must sort sequences by decreasing length before packing. This can improve performance slightly and is sometimes required in older codebases. The tradeoff is bookkeeping. You need to store the permutation used for sorting so you can restore the original order after the RNN if downstream code expects it.
If you are a beginner, prefer enforce_sorted=False unless you have a reason to manage sorting explicitly.
Practical Guidance: Choosing Between Packing and Masking
Packing is most helpful when using RNNs and when sequence lengths vary a lot. It saves computation on padded timesteps and ensures hidden states correspond to real ends of sequences.
Masking is required for losses and metrics and is the default approach for non RNN architectures. Even in RNN workflows, masking is still common at the loss stage.
A good mental model is: packing affects the forward computation inside the RNN, masking affects how you compute objectives and evaluation so padding does not matter.
Common Mistakes to Watch For
One frequent mistake is computing the “last timestep” output by indexing out[:, -1, :] on a padded tensor. That selects the last padded position for shorter sequences. If you are not using packing, you must gather the last valid timestep using lengths, typically by indexing at lengths - 1.
Another common mistake is padding with a value that is also a valid token id, which makes it impossible to distinguish padding from real content. Always reserve a dedicated padding id for discrete tokens.
Finally, be careful when moving data to devices. The padded tensor and the mask should be on the same device as the model. Length tensors used for packing are commonly kept on CPU, but any mask used in loss computations should be on the model device.
Do not take the last timestep with -1 indexing on padded batches unless every sequence has the same length. Use packing or gather using lengths - 1.