Table of Contents
What Makes Sequence Problems Different
Sequence modeling is about data where order matters. A sentence, an audio waveform, and a clickstream are all sequences because each element depends on what came before and sometimes what comes after. Compared to structured tabular inputs where you typically treat features as an unordered set, sequence inputs require you to decide what the “time” or “position” axis is, how to represent each step, and how to handle sequences that do not all have the same length.
A practical way to think about sequence tasks is to identify what goes in and what comes out. Sometimes you have many inputs and one output, like sentiment classification of a whole review. Sometimes you have many inputs and many outputs, like tagging each word with a part of speech. Sometimes you generate a sequence from another sequence, like translation. The rest of the chapters in this section will show the mechanics for these cases, but the unifying idea is that you process ordered steps and must be careful about padding, masking, and what you consider the “final” representation.
Core Shapes and Conventions You Will See
In PyTorch, sequence tensors often have three dimensions: batch size, sequence length, and feature size. A common layout is $(B, T, D)$, where $B$ is the number of sequences in the batch, $T$ is the number of time steps or tokens, and $D$ is the embedding or feature dimension at each step. Some PyTorch sequence modules also support a time first layout $(T, B, D)$, which you control with a parameter like batch_first=True in many recurrent layers.
It is important to explicitly track which axis is which, because a silent shape mismatch can produce training that runs but learns the wrong thing.
Rule: Always write down the intended tensor shape at the boundary of your model, at the output of your embedding or feature encoder, and at the output head. If you cannot state these shapes, debugging will be slow and errors will be subtle.
The High Level Modeling Pattern for Sequences
Most sequence models for beginners follow the same high level pipeline. First you turn raw inputs into per step vectors, for example using an embedding table for tokens or a small feature network for per time step measurements. Next you pass the sequence of vectors through a sequence backbone that can mix information across time, such as an RNN, LSTM, GRU, or a Transformer. Finally you map the backbone output to the target format using a head, such as a linear layer for classification.
The key decision is what representation to send to the head. For many to one tasks, you might use the final hidden state, or a pooled summary over time. For many to many tasks, you usually apply the head at every time step. For sequence to sequence tasks, you often have an encoder and a decoder, where the decoder generates outputs step by step.
Variable Length Sequences, Padding, and Masks
Real datasets rarely have a fixed sequence length. The standard approach is to pad shorter sequences up to the length of the longest sequence in the batch so that you can stack them into a single tensor. Padding introduces fake elements that must not contribute to the loss or to certain computations.
There are two common mechanisms to deal with padding. One is to keep a lengths tensor, often shaped $(B,)$, that stores the true length of each sequence before padding. Another is to build a boolean mask, often shaped $(B, T)$, that indicates which time steps are real and which are padding. Different backbones and losses use one or the other.
Rule: If you pad sequences, you must ensure that the padding does not affect the loss. For token level losses, use an ignore index for padded targets or explicitly mask the loss before averaging.
Choosing a Backbone in This Section
This section focuses on recurrent models like RNNs, LSTMs, and GRUs, which process sequences step by step and maintain a hidden state. They are conceptually simple and are a good starting point for beginners because they work with smaller datasets and smaller compute budgets. Later in the course, the Attention and Transformers section will introduce self attention models that parallelize over time more easily and dominate many modern NLP tasks.
Even if you plan to use Transformers, learning recurrent models first helps you build intuition for hidden state, temporal dependencies, and sequence loss handling.
Typical Tasks You Will Build Toward
A basic text classification pipeline takes a sequence of tokens and outputs a single class label. This typically uses an embedding layer, a recurrent backbone, and a classifier head. The model must handle variable length inputs and usually uses padding and an ignore rule for padded positions.
Sequence labeling produces an output for every input step, like tagging each token. This typically applies a linear head to every hidden state, then computes a per time step loss while ignoring padding.
Sequence to sequence modeling maps one sequence to another, like translation. The simplest version uses an encoder to summarize the input and a decoder to generate output tokens, often using teacher forcing during training. Because generation is stepwise, you must keep track of start and end tokens and stop conditions.
How to Think About Losses for Sequences
Sequence learning often uses cross entropy at each time step, then aggregates across time and across the batch. If you have a per step loss $\ell_{b,t}$, a common objective is the mean over all non padded positions:
$$
L = \frac{1}{\sum_{b=1}^{B}\sum_{t=1}^{T} m_{b,t}} \sum_{b=1}^{B}\sum_{t=1}^{T} m_{b,t}\,\ell_{b,t},
$$
where $m_{b,t}$ is 1 for real tokens and 0 for padding.
Rule: Do not average the loss over padded positions. If you do, the model is partially trained to predict padding behavior, and metrics will be misleading.
What You Should Be Ready to Implement After This Section
By the end of the Recurrent and Sequence Models section, you should be able to take a dataset of variable length sequences, batch it with padding, run it through an RNN style model in PyTorch, compute a masked loss, and evaluate correctly. You should also be able to distinguish tasks by their input and output shapes, choose whether your head operates on the final state or all states, and reason about how teacher forcing changes training dynamics for generation.