Table of Contents
Where Attention Fits in Deep Learning
Many problems involve inputs that are naturally sequences, such as words in a sentence, frames in audio, or events over time. Earlier sequence models process tokens in order, which can make it harder to connect information that is far apart. Attention addresses this by letting the model choose, for each position, which other positions to focus on when building a representation.
At a high level, attention produces a new sequence of vectors where each output position is a weighted mixture of input positions. The weights are learned dynamically from the input itself, which is why attention is often described as content based routing.
Key idea: attention computes weights over a set of vectors and returns a weighted sum. The weights depend on similarity between a query and keys, and the values are what gets averaged.
The Core Attention Computation
In the most common form used in transformers, each token embedding is projected into three vectors called query, key, and value. Collect them into matrices $Q$, $K$, and $V$. Scaled dot product attention is:
$$
\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V
$$
Here $d_k$ is the key dimension, and $M$ is an optional mask that blocks certain positions by adding a very negative number before the softmax.
The term $QK^\top$ creates a similarity score between every query position and every key position. Softmax turns those scores into weights that sum to 1 for each query position, and multiplying by $V$ averages the value vectors using those weights.
Important rule: the softmax is applied across the key dimension for each query, so each output token is a convex combination of value vectors.
Self Attention Versus Cross Attention
Self attention means $Q$, $K$, and $V$ come from the same sequence. Each token can attend to other tokens in that same input. Cross attention means queries come from one sequence while keys and values come from another, which is common in encoder decoder setups where a decoder attends to encoder outputs.
For beginners, the practical distinction is mostly about tensor shapes and where $Q$ is computed from. The same formula applies, only the source of the matrices changes.
Causal Masks and Padding Masks
Two masks show up constantly in transformer models.
A padding mask prevents the model from attending to padded tokens that were added just to make all sequences in a batch the same length. Without this mask, padding can leak into representations and hurt training.
A causal mask enforces left to right generation by blocking attention to future tokens. This is essential for autoregressive language modeling. Conceptually, it creates a triangular pattern where position $t$ can only attend to positions $\le t$.
If you are training a model that predicts the next token, you must apply a causal mask, otherwise the model can peek at the answer.
Multi Head Attention
Multi head attention runs attention several times in parallel with different learned projections. Each head has its own $W_Q$, $W_K$, and $W_V$ matrices, producing multiple attention outputs that are concatenated and projected back to the model dimension.
If the model dimension is $d_\text{model}$ and there are $h$ heads, a common choice is per head dimensions $d_k = d_v = d_\text{model} / h$. Each head can specialize, for example focusing on local context, syntax like relationships, or longer range dependencies.
The main benefit is representational capacity, not a change in the core math.
Transformer Blocks at a Glance
A transformer is built from repeated blocks. Each block contains an attention sublayer and a position wise feedforward network, plus residual connections and normalization.
The feedforward network is applied independently to each position and typically has the form:
$$
\mathrm{FFN}(x) = W_2 \,\sigma(W_1 x + b_1) + b_2
$$
where $\sigma$ is a nonlinearity such as GELU or ReLU, and the hidden dimension is larger than $d_\text{model}$.
Residual connections add the sublayer input back to its output. Normalization is commonly LayerNorm. Dropout is often used in attention weights and in the feedforward network.
Do not forget residual connections and normalization in transformer blocks. Attention alone is not a full transformer layer.
Positional Information
Attention by itself does not know the order of tokens. If you shuffle the tokens, the set of embeddings is the same and attention has no built in notion of position. Transformers add positional information to token embeddings so that the model can represent order.
Two common approaches are fixed sinusoidal positional encodings and learned positional embeddings. The key point for beginners is that position information is injected into the input representations before attention is applied, so attention scores can indirectly depend on position.
Typical Shapes You Will See
In PyTorch, you will often represent a batch of token embeddings as a tensor of shape $(B, T, d_\text{model})$, where $B$ is batch size and $T$ is sequence length. Attention computations internally build score tensors shaped like $(B, h, T, T)$ for self attention, because every query position attends to every key position for each head.
This has two practical consequences. First, memory and compute grow roughly like $T^2$, which is why long sequences are expensive. Second, masks must broadcast to match these score shapes, which is a frequent source of shape bugs.
Transformer self attention has $O(T^2)$ attention scores. Long sequences can quickly become a memory bottleneck.
What This Chapter Sets You Up For
This chapter established the mental model and core equations behind attention, self attention, masking, multi head attention, and the high level transformer block structure. Next, you will use these ideas to understand the motivation for attention, build intuition for self attention, and then work with transformer modules and pretrained transformer models in PyTorch, including tokenization and batching details.