11.2. Self-Attention Intuition

Table of Contents

From Recurrence to Direct Pairwise Interaction

Self attention is a way for a model to decide which parts of an input sequence matter for each position in that sequence. If you have tokens $x_1, x_2, \dots, x_n$, self attention lets the representation at position $i$ look at every position $j$ and form a weighted combination of what it finds useful. The key intuition is that the model is not forced to process information only step by step, it can connect distant tokens in a single operation by assigning them high weight.

A helpful mental model is, each token asks a question about what it needs, then scans all tokens for answers, then mixes the best answers into an updated token representation.

Queries, Keys, and Values as Roles

Self attention uses three learned projections of the same input token representations. For each token vector $x_i$, the model produces a query $q_i$, a key $k_i$, and a value $v_i$. You can think of the query as describing what this position is looking for, the key as describing what each position offers, and the value as the information that will actually be gathered if that position is attended to.

All tokens produce keys and values, and each token produces its own query. Then each query is compared to all keys to get relevance scores. High relevance means, pay more attention to that token.

Attention Scores and Weighted Averaging

For a fixed position $i$, self attention computes a score between $q_i$ and every key $k_j$. The standard score is a dot product. Larger dot products mean more alignment, and therefore more attention.

Those scores are then normalized into weights that sum to 1 using softmax, and the output at position $i$ becomes a weighted sum of the value vectors:

$$
\alpha_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \quad
y_i = \sum_{j=1}^{n} \alpha_{ij} v_j.
$$

Here $d_k$ is the dimensionality of the keys and queries. The division by $\sqrt{d_k}$ is a scaling that keeps the dot products from becoming too large as the dimensionality grows, which helps training stay stable.

Important rule: for each position $i$, the attention weights $\alpha_{ij}$ form a probability distribution over $j$, so $\sum_{j=1}^{n} \alpha_{ij} = 1$.

What the Model Gains From This

Self attention gives two practical benefits in one mechanism. First, it provides dynamic, input dependent weighting. The same token can attend to different other tokens in different sentences, depending on context. Second, it builds contextual representations. After attention, the representation at each position contains information pulled from other positions, not just from the token itself.

A classic intuition is pronoun resolution. In a sentence like “The animal did not cross the street because it was tired,” the token “it” should gather information from “animal.” Self attention allows the “it” position to put high weight on the “animal” position, so the resulting representation of “it” carries features of what it refers to.

Matrix View, Same Idea but Faster

Although it is useful to picture a single token attending to others, the computation is usually done for all tokens at once using matrices. If you stack all queries into a matrix $Q$, all keys into $K$, and all values into $V$, then the attention weights are computed as:

$$
A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right), \quad
Y = AV.
$$

The core intuition stays the same. Each row of $A$ corresponds to one token’s attention distribution over all tokens.

Important shape check: if the sequence length is $n$, then $A$ has shape $n \times n$. Each row attends across the full sequence.

Why It Is Called Self Attention

It is called self attention because the queries, keys, and values all come from the same sequence. The sequence attends to itself. This is different from cross attention, where queries come from one sequence and keys and values come from another, which you will see when discussing encoder decoder style models.

Causality and Masking Intuition

In some tasks, like language modeling, the token at position $i$ must not look at future tokens $j > i$. The same attention idea works, but with a mask that prevents attention to forbidden positions by forcing their scores to behave like $-\infty$ before softmax, making their weights effectively zero.

Important rule for causal settings: when predicting token $i$, attention must not use information from positions $> i$, otherwise training leaks future information and evaluation becomes misleading.

A Simple Way to Think About It While Coding

When you later implement self attention in PyTorch, keep one guiding intuition in mind. Each output token is a learned mixture of input token information, where the mixture weights are computed from similarity between a learned “what I need” vector and learned “what I offer” vectors. If that picture is clear, the formulas and the tensor operations will feel like a direct translation rather than new concepts.