Table of Contents
What a Sequence-to-Sequence Model Is
A sequence-to-sequence model, often shortened to seq2seq, maps one sequence to another sequence, where the output length can be different from the input length. Classic examples are machine translation, where an English sentence becomes a French sentence, summarization, where a long text becomes a shorter text, and dialogue generation, where a user message becomes a response. In contrast to simple classification, the model must produce a whole series of tokens, one step at a time, and each step depends on what has been produced so far.
The Encoder and Decoder Idea
The traditional seq2seq design has two parts, an encoder and a decoder. The encoder reads the input tokens in order and produces a representation of the input. The decoder then generates the output tokens one by one using that representation plus its own internal state.
In the simplest RNN based setup, the encoder processes $x_1, x_2, \dots, x_T$ and ends with a final hidden state $h_T$. The decoder is initialized from that state and generates $y_1, y_2, \dots, y_U$. At each decoding step $t$, it predicts a probability distribution over the vocabulary for the next token.
A common factorization is:
$$
p(y_{1:U} \mid x_{1:T}) = \prod_{t=1}^{U} p(y_t \mid y_{<t}, x_{1:T})
$$
This formula is the basic reason decoding is sequential, each prediction depends on all earlier predicted tokens.
Special Tokens: SOS, EOS, and Padding
Seq2seq pipelines typically add a start of sequence token, often written as SOS, and an end of sequence token, often written as EOS. The decoder usually starts by receiving SOS as its first input, then predicts the next token. Generation stops when EOS is produced or a maximum length is reached. Padding tokens are used so sequences in a batch can share a common length, but padding should not contribute to loss.
Always mask padding in the loss for seq2seq tasks. If you compute loss on PAD positions, the model learns to predict PAD and training quality degrades.
Teacher Forcing Versus Free Running
During training, the decoder can be fed the ground truth previous token instead of its own previous prediction. This is called teacher forcing. It makes optimization easier because the decoder sees correct history. During inference, ground truth is not available, so the decoder must use its own generated tokens. This mismatch between training and inference is a common source of errors, such as repetitive outputs or drifting off topic after a few steps.
You do not need to implement a full teacher forcing strategy here, but it is important to understand that training and inference use different inputs to the decoder loop.
Do not evaluate a seq2seq model by feeding ground truth tokens at inference time. That measures a different behavior than real generation.
Output Layer and Loss Shape
At each decoding step, the model outputs logits over the vocabulary, typically shaped like $(B, U, V)$ for batch size $B$, output length $U$, and vocabulary size $V$. Targets are usually integer token ids shaped $(B, U)$. Loss is commonly token level cross entropy averaged over non padded positions.
A typical pattern is to flatten time and batch for the loss, treating all predicted positions as one big set of classification problems:
logits reshaped to $(B \cdot U, V)$ and targets reshaped to $(B \cdot U)$, then apply a loss that ignores PAD via an ignore index.
Inference: Greedy Decoding and Beam Search
At inference, you choose tokens based on the predicted distribution. The simplest approach is greedy decoding, which picks the most likely token at each step. Greedy is fast but can miss better global sequences.
Beam search keeps multiple partial hypotheses at once. At each step, it expands candidates and retains the top $k$ by score, where $k$ is the beam width. Beam search often improves quality for tasks like translation, but it is slower and more complex.
When using beam search, compare sequences using summed log probabilities, not raw probabilities. Raw probabilities underflow and prefer shorter sequences incorrectly.
The Limitation of the Simple Encoder Final State
If the decoder only receives the final encoder state, the encoder must compress the entire input into a single vector. This becomes a bottleneck for long sequences. This limitation motivates attention mechanisms, which allow the decoder to look back at different encoder states during generation. The detailed attention and transformer treatment belongs elsewhere, but the key takeaway here is that basic RNN seq2seq works best on shorter sequences, or simple tasks, unless you add attention.
A Minimal PyTorch Skeleton for the Idea
In practice, you typically have an encoder module and a decoder module, each built from an RNN, LSTM, or GRU. The training forward pass often looks like: embed input tokens, run the encoder, then run the decoder across the target length to produce logits. The decoding loop is explicit during inference and sometimes explicit during training.
Conceptually, the decoder loop is:
Initialize decoder hidden from encoder output. Set current token to SOS. Repeat: compute next token distribution, pick or sample the next token, feed it back in, stop at EOS.
Even if you implement the decoder in a vectorized way during training, you should still understand this step by step process because inference is inherently sequential.
Practical Rules of Thumb for Beginners
Seq2seq problems are easy to break with small bookkeeping mistakes. You will save time by consistently checking that your input and target sequences are shifted correctly. The decoder should be trained to predict the next token, so decoder inputs are typically the target sequence shifted right with SOS prepended, and the prediction targets are the original target sequence with EOS appended, both aligned in time.
Align decoder inputs and targets correctly. If the model is trained to reproduce its input token at the same position instead of predicting the next token, it can appear to train while failing at generation.
What to Take Forward
A seq2seq model generates an output sequence one token at a time, using an encoder representation of the input and a decoder conditioned on previous tokens. Training commonly uses teacher forcing and a masked token level cross entropy loss. Inference uses greedy or beam search and stops at EOS. The simple encoder final state approach is a useful starting point, but attention is the standard upgrade for longer or more complex sequences.