10.1 Sequences, Tokens, and Padding

What a Sequence Is in Deep Learning

A sequence is any ordered list where position matters. In practice, sequences show up as text characters or words, time series readings, event logs, and clicks. Unlike a fixed size feature vector, a sequence has a length, and that length can vary from example to example. PyTorch models that handle sequences usually expect tensors that are rectangular, so the first practical problem you face is how to represent variable length examples in a single batch.

Tokens and Tokenization

Neural networks do not consume raw text. They consume numbers. Tokenization is the step that turns text into a sequence of discrete symbols called tokens. A token might be a character, a word, or a subword unit, depending on the tokenizer you choose. For beginners, it is enough to understand that tokenization produces a list of token ids, which are integers that index into a vocabulary.

A vocabulary is a mapping from token to integer id. If your vocabulary size is $V$, then token ids are typically in the range $0$ to $V-1$. The same sentence tokenized with different schemes can produce different lengths, which directly affects padding and batching later.

Special Tokens You Will Almost Always Need

Most sequence pipelines reserve a few ids for special tokens. The exact names vary, but the ideas are consistent.

A padding token, often written as [PAD], fills unused positions so that shorter sequences can be stacked with longer ones in the same batch. An unknown token, often [UNK], represents any token that is not in your vocabulary. Many setups also use a beginning of sequence token and an end of sequence token, often [BOS] and [EOS], which explicitly mark boundaries. Even when you do not use BOS and EOS, you must still be consistent about how you represent the start and end of the content, especially for generation tasks later.

Turning Token Lists Into Tensors

After tokenization you have Python lists like [12, 5, 932, 77]. A single example can become a 1D tensor of dtype torch.long, because token ids are indices.

When batching, you need a 2D tensor. The two most common layouts are batch first with shape (batch_size, seq_len) and time first with shape (seq_len, batch_size). In modern PyTorch code, batch first is common because it matches how many other layers expect data, but some sequence modules historically used time first. What matters is that you choose one convention and stick to it through your DataLoader and model forward pass.

Why Padding Is Necessary for Batching

A DataLoader batch is usually a single tensor. If one sentence has length 4 and another has length 9, you cannot stack them into a single 2D tensor unless you make them the same length. Padding solves this by extending shorter sequences with [PAD] until they match the longest sequence in the batch.

If the longest sequence length in a batch is $L$, and the $i$th sequence has length $\ell_i$, then you append $L - \ell_i$ pad tokens to that sequence. The resulting batch tensor has shape (B, L).

Padding can be added on the right, called right padding, or on the left, called left padding. Right padding is common for many RNN style setups because the real content starts at index 0. Left padding is common in some transformer tokenizers for historical and practical reasons. The key is to match whatever your downstream model and masking logic expects.

Attention Masks and Padding Masks

Once you pad, the model must not treat padding as real content. There are two common ways to communicate this.

One way is to provide lengths, a 1D tensor of true sequence lengths (B,), which some sequence utilities can use to ignore padded positions. Another way is to provide a mask, usually a boolean or 0/1 tensor of shape (B, L) that indicates which positions are real tokens.

A typical attention mask uses 1 for real tokens and 0 for padding. If input_ids is (B, L), then attention_mask is also (B, L). Even if you are not using attention yet, thinking in terms of masks is helpful because it generalizes across model families.

Padding and Loss Computation for Sequence Tasks

For tasks where you predict a label per sequence, like sentiment classification, padding mainly affects the encoder that reads the sequence, and you still have one target per example. For tasks where you predict a label per token, like language modeling or tagging, padding directly affects the loss because you have targets at each time step.

In token level tasks, you typically flatten predictions and targets across batch and time, but you must exclude pad positions. Many PyTorch loss functions support this using an ignore index. Conceptually, you set the target at padded positions to a special value and configure the loss to ignore it.

Truncation and Maximum Sequence Length

Sometimes sequences are too long to fit in memory or to be efficient. Truncation is the act of cutting sequences to a maximum length $L_{\max}$. This is not just a performance detail, it changes the problem because you may drop information. You should choose $L_{\max}$ intentionally based on your data distribution and task needs.

You can truncate from the end, from the beginning, or using a windowing strategy, depending on whether early or late tokens are more important. Whatever you choose, apply it consistently and document it, because it affects reproducibility and evaluation.

A Minimal PyTorch Style Example of Padding a Batch

Suppose you already have token id lists for a batch, for example [[4, 9, 2], [8, 1, 7, 7, 3]]. You choose a pad_id, such as 0. You compute the max length in the batch, pad each list to that length, and convert to a torch.long tensor of shape (B, L). Alongside it, you build a mask that is 1 for the original tokens and 0 for the padded positions. That pair, ids and mask, is the standard output of many text DataLoaders.

The details of implementing this live in your collate function and DataLoader setup, which is covered elsewhere in the course. The important takeaway here is the representation: padded input_ids plus a mask or lengths that tells the model what is real.

Common Beginner Mistakes to Avoid

A frequent mistake is padding with a token id that is also a real token, which makes it impossible for the model to distinguish padding from content. Another mistake is mixing left and right padding between training and inference, which effectively shifts where the model expects information. A third mistake is forgetting to move masks to the same device as inputs when training on GPU, which leads to device mismatch errors when the mask participates in computation.