Attention, from the bottleneck up

A translation model reads an English sentence and writes a French one. Give it something short and it does fine. Give it a sixty-word sentence and the output starts confidently, drifts somewhere around the middle, and finishes by inventing a clause that was never there. It has not run out of vocabulary. It has not forgotten French grammar. The first half of the output proves both.

It has run out of somewhere to put the sentence.

By the end of this you will know exactly what that means, why the repair could only have taken one shape, why there is a square root in the denominator of the equation everyone recites, and what happens if you delete it. Queries, keys and values do not appear until the very end, and only as a packaging decision made three years later. This post is about the idea.

The machine that had to remember everything#

In 2014 two groups landed on the same architecture within months of each other: Sutskever, Vinyals and Le, and Cho and colleagues. One recurrent network reads the source sentence. Another writes the target. The first is the encoder, the second the decoder, and between them passes a single vector.

The encoder is a loop:

$$h_t = f(x_t,\, h_{t-1}), \qquad c = h_T$$

$x_t$ is the embedding of the $t$-th source word. $h_t$ is the hidden state after reading it, a vector of some fixed size, typically 1000 numbers. $f$ is an LSTM or GRU cell, which for our purposes is a learned function that takes the previous state and the current word and produces the next state. $h_T$ is the state after the last word. And $c$, the context vector, is defined to be exactly that: the final hidden state, nothing more.

The cell is worth one sentence of respect, because it was itself a fix for this same family of problem. A plain recurrent network multiplies its state by a matrix at every step, so gradients flowing back through forty steps get multiplied forty times and either vanish or explode. The LSTM adds a separately maintained cell state and learned gates that decide what to keep, what to overwrite and what to read out, which gives the gradient a path along the sequence that is closer to addition than to repeated multiplication. It made long sequences trainable. It did not make the state any larger.

The decoder then generates one word at a time:

$$p(y_i \mid y_1,\dots,y_{i-1},\, \mathbf{x}) = g(y_{i-1},\, s_i,\, c)$$

$y_i$ is the $i$-th output word, $s_i$ the decoder’s own hidden state, $g$ another learned function, and $\mathbf{x}$ the whole source sentence, which reaches the decoder only through $c$. During training the decoder is fed the true previous word rather than its own guess, so the sentence is learned in parallel across positions; at inference it consumes what it just produced, which is why an early mistake propagates.

Look at the subscripts. $y$ has one. $s$ has one. $c$ does not.

Every French word is generated from the same fixed vector. The word “signed” at the end of the English sentence and the word “the” at the beginning both had to be packed into those same thousand numbers, and every step of the output has to unpack whatever it needs from that one package.

This deserves to be said sympathetically, because it was a good design. Sutskever’s system scored 34.8 BLEU on the WMT’14 English to French task against 33.3 for the phrase-based statistical system it was competing with, and 36.5 when it was used to rerank that system’s output. A neural network beat a decade of hand-built machine translation infrastructure. At ten or fifteen words, one vector is enough. It was not a design with a flaw. It was a design with a ceiling that nobody had yet walked into.

The tell#

Here is the detail that gives the whole thing away, and it is in Sutskever’s abstract.

They found that reversing the word order of the source sentence, but not the target, improved performance “markedly”. Feed the model the English backwards. Train it to produce French forwards. It gets better.

Stop and take that seriously, because it makes no sense. The information content of a reversed sentence is identical. Nothing was added. Nothing was removed. A model that understood the sentence would not care which end it started from.

The answer is about distance. Read forwards, the first English word is separated from the first French word by the entire sentence: the encoder has to carry it through every intermediate state before the decoder ever starts. Read backwards, the first English word is the last thing the encoder sees, so it sits right next to the moment the decoder needs it. The later words pay for this, but apparently the early ones matter more, because getting the beginning of the output right keeps the rest on track.

The authors’ own explanation is that reversal creates short-term dependencies between source and target, which makes the optimisation easier. Same observation, stated as a gradient problem rather than a memory one.

This is worth naming as a general move, because it keeps working:

When a hack helps for a reason nobody can quite justify, the hack is telling you where the architecture hurts.

A trick that only pays off if information decays over distance is evidence that information decays over distance.

One honesty note, and it matters. This was an empirical finding first and a theory second. They tried it, it worked, and the explanation came afterwards, offered tentatively. It is a good story and it is very probably right, but do not let anyone tell you the reasoning came first. It did not.

Naming the bottleneck#

By late 2014 the failure had a measurement. Cho and colleagues took the encoder-decoder apart and reported, in their own words, that its performance “degrades rapidly as the length of the sentence and the number of unknown words increase”. Plot BLEU against source sentence length and the curve is flat for a while and then falls off a cliff. Short sentences fine. Long sentences not fine. The transition is not gentle.

Now the capacity argument, which is the intuition everyone reaches for.

$c$ is a vector of fixed dimension $d$. In float32 that is $32d$ bits. It is $32d$ bits when the source sentence is four words and it is $32d$ bits when the source sentence is eighty. A 1000-dimensional context vector is 32,000 bits, always, forever. Meanwhile the space of things an eighty-word sentence could be saying is vastly larger than the space of things a four-word sentence could be saying. Something has to be thrown away.

But the throwing away is not the worst part. This is:

The encoder has to decide what to discard before it knows what the decoder will need.

It is being asked to write a summary for a reader whose questions it has not heard. And it has to write the same summary regardless of which question comes.

There are actually two different failures being described here, and they get conflated constantly, including by me two paragraphs ago.

One is capacity: a fixed vector cannot hold an unbounded amount of sentence, so at some length information is necessarily destroyed. This is an argument about the forward pass, and it would apply even to a perfectly trained model.

The other is optimisation: gradients have to travel back along the whole recurrence to teach the encoder what the decoder needed, and over forty steps that signal degrades. This is an argument about training, and it is the one the reversal trick speaks to, since reversal does not change capacity by a single bit.

Both are real, they produce the same symptom, and the repair in the next section happens to fix both at once: keeping every encoder state removes the capacity ceiling, and connecting each output step directly to each input step gives the gradient a short path. That the same move solves two distinct problems is a large part of why it worked as well as it did.

The mandatory honesty note. The bit-counting argument is intuition, not proof. Real sentences are nowhere near uniformly distributed over the space of token sequences, neural representations are not remotely efficient codes, and nobody has derived a bound saying ”$d = 1000$ fails at $n = 40$”. What we actually have is a clean intuition and a curve that bends in the place the intuition says it should. This is the explanation everyone gives, it is almost certainly the right shape, and it is not a proof.

The obvious fix, if you let yourself have it#

Bahdanau, Cho and Bengio named it directly. Their abstract conjectures that the fixed-length vector is “a bottleneck” in improving the architecture. The framing in this post is theirs, not something retrofitted afterwards.

Now derive the repair. Do not look it up: each step is forced.

One. The problem is that we computed $h_1, \dots, h_T$ and then threw all of them away except the last. So stop throwing them away. Keep all of them. Memory is cheap and we already paid the compute.

Two. Now the decoder has $T$ vectors available and needs one vector to condition on. It has to choose.

Three. Choosing hard, picking an index $j$ and using $h_j$, is not differentiable. The gradient cannot flow back through an argmax to teach the model which index it should have picked. Backpropagation dies here. Reject.

Four. So the choice has to be soft: a weighted average of all $T$ vectors. And the weights have to depend on what the decoder is generating right now. Both halves of that sentence are load-bearing, and it is worth killing the two obvious alternatives to see why.

Average them uniformly. Differentiable, simple, and useless: every output step receives the same mixture, which is a fixed-size summary of the whole sentence computed without knowing the question. That is the thing we are trying to get away from, now with worse resolution.

Use a fixed window around the current position. Better, and it encodes a real prior, that translation is roughly monotone. But it needs a rule for where the window goes, and the moment you try to write that rule down you are back to choosing a position based on the decoder state. The choice problem does not go away, it just gets hidden inside a heuristic that will be wrong for any language pair that reorders.

Five. What should the weights look like? Non-negative, because subtracting one source word’s meaning from the mixture has no sensible reading. And summing to one, so that the magnitude of the result does not grow with sentence length, and so the weights can be read as “how much of each source position went into this”.

Six. Non-negative, sums to one, differentiable, computed from arbitrary real-valued scores. That is the softmax. It was not chosen for elegance. It is what those four constraints leave you with.

And the context vector gets its subscript back:

$$c \;\longrightarrow\; c_i$$

That is the whole repair. Every step of the output builds its own summary of the input.

The mathematics, in three pieces#

Score, normalise, mix. Every attention mechanism ever published is these three steps, and they differ only in the first one.

Score#

$$e_{ij} = a(s_{i-1},\, h_j)$$

Read it as: how relevant is source position $j$ to what I am about to generate at output step $i$. $s_{i-1}$ is the decoder state just before generating token $i$, $h_j$ is the encoder state at source position $j$, and $a$ is a learned scoring function. This runs for every pair, so a 20-word source and a 25-word target means 500 scores.

That product is the price of the repair, and it is the number that governs everything that happened afterwards. Cost grows as source length times target length, where the old architecture paid for the source once. Nobody minded in 2014, when sentences were the unit of work. It became the central engineering problem of the field about five years later, when the unit of work became a document.

Note also what the score function is allowed to see: the decoder state $s_{i-1}$, which carries everything generated so far, and the encoder state $h_j$, which carries the source around position $j$. It does not see $j$ or $i$ directly. Any sense of position it has must already be inside those vectors, which for a recurrent encoder it is, because the state was built by reading the sentence in order. Remove the recurrence and that stops being true, which is exactly the problem positional encodings are invented to solve later.

Bahdanau’s choice was a small neural network:

$$e_{ij} = v_a^{\top} \tanh\!\left(W_a s_{i-1} + U_a h_j\right)$$

$W_a$ and $U_a$ are learned matrices that project the decoder state and the encoder state into a shared space, $\tanh$ squashes their sum, and $v_a$ is a learned vector that collapses the result down to a single number. It is a one-hidden-layer network evaluated once per pair.

A year later Luong, Pham and Manning pointed out you could just use the dot product:

$$e_{ij} = s_i^{\top} h_j$$

No parameters at all. This is allowed to work because the dot product is unnormalised cosine similarity: if the encoder and the decoder learn to place related content in aligned directions, then the dot product already measures relevance, for free.

What breaks if you swap additive for dot product. The dot product requires $s$ and $h$ to have the same dimension and to live in comparable spaces. Additive scoring does not, since $W_a$ and $U_a$ can map from anywhere to anywhere. In exchange, the dot product is one matrix multiply across all positions at once instead of an MLP evaluated per pair. Worth being honest about why it won: Luong and colleagues found the two competitive in quality, and dot product’s eventual dominance is at least as much about what a GPU is fast at as about modelling.

Normalise#

$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T} \exp(e_{ik})}$$

The scores become a distribution over source positions. This is the equation people skim, so here are the three separate things that break without it.

1. The weights no longer sum to one, so the scale of $c_i$ drifts with sentence length. Sum sixty roughly-unit-weighted vectors instead of six and the result is several times the magnitude. The decoder sees an input distribution it was never trained on, and it sees a different one for every sentence length.
2. Negative weights would subtract encoder states from the mixture. There is no reading of “minus 0.3 of the word agreement” that means anything.
3. You lose the interpretation as a distribution over source positions, which is the only reason the heatmaps in every attention paper are meaningful at all.

Mix#

$$c_i = \sum_{j=1}^{T} \alpha_{ij}\, h_j$$

What breaks without it. This is the step that makes attention soft. Replace the weighted sum with $h_{\arg\max_j \alpha_{ij}}$, just take the best one, and you have hard attention: not differentiable, trainable only with reinforcement learning or a relaxation, and much worse behaved.

It is tempting to read soft attention as a convenient approximation of the hard version you really wanted. That is backwards. Soft attention is the version that fits inside backpropagation, and fitting inside backpropagation is the entire reason it took over.

The square root#

Here is the equation everyone can recite, from Vaswani and colleagues:

$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

Ignore $Q$, $K$ and $V$ for now: that is a packaging decision and it belongs to a later post. The part that matters here is the division, and the paper’s stated reason for it. They suspect that for large $d_k$ the dot products grow large, pushing the softmax into a region where its gradients are extremely small.

Almost every explanation stops there. The derivation is four lines and it is worth seeing.

Assume the components of $q$ and $k$ are independent, mean 0, variance 1. Then for a single component:

$$\mathbb{E}[q_i k_i] = \mathbb{E}[q_i]\,\mathbb{E}[k_i] = 0$$ $$\mathrm{Var}(q_i k_i) = \mathbb{E}[q_i^2 k_i^2] - 0 = \mathbb{E}[q_i^2]\,\mathbb{E}[k_i^2] = 1$$

and since the $d_k$ component products are independent, their variances add:

$$\mathrm{Var}(q \cdot k) = \sum_{i=1}^{d_k} \mathrm{Var}(q_i k_i) = d_k$$

So the standard deviation of the raw dot product is $\sqrt{d_k}$. It grows with the dimension you are scoring in. At $d_k = 64$, the base transformer’s per-head dimension, the scores spread over roughly plus or minus 8. At $d_k = 512$, roughly plus or minus 23.

I ran it to check the arithmetic rather than trusting it:

$d_k$	$\sqrt{d_k}$	measured SD of $q \cdot k$
8	2.83	2.82
64	8.00	7.98
512	22.63	22.61

Now feel the consequence. A softmax over scores that are twenty apart is one-hot to within floating point. And the softmax Jacobian is

$$\frac{\partial p_i}{\partial e_j} = p_i(\delta_{ij} - p_j)$$

where $\delta_{ij}$ is 1 when $i = j$ and 0 otherwise. Every term has a factor of $p_i$ or $(1 - p_j)$, so the whole thing goes to zero when any probability approaches 0 or 1. Saturated softmax means no gradient, and no gradient means the scoring function stops learning. The model does not train badly. It stops training.

The dashed line is what the scaling buys: divide by $\sqrt{d_k}$, the scores return to unit variance, and the distribution looks the same whether you are scoring in 8 dimensions or 512. In that simulation the scaled softmax spreads its mass over about 41 of the 64 positions at every dimension. Unscaled at $d_k = 512$ it spreads over 1.01.

Two honesty notes, both required.

First, the variance argument assumes mean 0, variance 1, and independent components. That describes a network at initialisation. It does not describe a trained one, where $q$ and $k$ are outputs of learned projections and are neither independent nor unit variance. The scaling is kept anyway, and the justification for keeping it is that models train better with it. That is an empirical argument wearing a theoretical costume.

Second, read the paper’s own wording again. They suspect. The most cited paper in the field hedged on this, and the hedge is still the honest position.

What attention is not#

It is not the model understanding anything. It is a learned, input-dependent weighted average. The word “attention” is a metaphor that has been load-bearing for a decade, and it has quietly convinced a lot of people that something cognitive is happening. A weighted average is happening.

Here is a real one. These are cross-attention weights pulled out of a small English to French translation model, last decoder layer, averaged over heads. Nothing is cropped.

One step at a timeWhole matrix

Pick a French token. The English tokens below light up in proportion to how much of it went into producing that step.

L'accordsurl'Espaceéconomiqueeuropéenaétésignéenaoût1992.</s>

TheagreementontheEuropeanEconomicAreawassignedinAugust1992.</s>

The content words do what the 2014 papers promised. accord takes 0.53 from agreement, Espace takes 0.73 from Area, août takes 0.74 from August. If you had only ever seen the figures in the papers, this is the part you would recognise.

Now look at the end-of-sentence token. The median output step puts 0.45 of its attention there, and one step puts 0.84. Every French function word does it: L’, l’, a, été, en all attend mostly to a token that carries no meaning at all.

That is not a bug, and it is not the model being confused. It follows from the normalisation in section 5. The weights are forced to sum to one, so a step that has nothing in particular to look at cannot simply attend to nothing. The mass has to go somewhere, and it collects on a semantically empty token. This is now usually called an attention sink, and once you know it is there you see it everywhere.

It also means the published alignment figures are cropped. Not dishonestly, and usually with the crop stated in the caption, but the picture you have in your head of what attention looks like is the content-word corner of a matrix where roughly half the mass is parked somewhere else.

Attention weights are not explanations. This is a live disagreement, and presenting it as settled in either direction would be dishonest.

Jain and Wallace showed that you can often construct a completely different attention distribution that produces the same prediction. If two different “explanations” yield identical output, neither can be the reason for it. Wiegreffe and Pinter pushed back, arguing the experimental setup does not support a conclusion that strong, and that it depends on what you wanted the word “explanation” to mean.

My position is that this is genuinely open, and that the safe reading of a heatmap is as a description of the computation, not as a justification of the output. The weights tell you what the model mixed. They do not tell you why the answer was right.

A heatmap is one slice of a much longer computation. Real models stack these layers, and each layer’s output is a mixture that the next layer mixes again. By the time a value from source position 3 has been through six layers it has been blended with everything else several times over, so “position 3 got weight 0.6” in layer one does not mean the output depended on position 3 to that degree. The arrow from a single layer’s weights to a claim about the finished prediction is much longer than a picture makes it look.

The alignment pictures are task-specific. Bahdanau’s famous figure looks like a word alignment because translation has a word alignment to find: French words really do correspond to English words. Do not carry that intuition into a language model, where there is no ground truth alignment for the picture to recover and a clean-looking map may be showing you nothing at all.

What this unlocked#

Notice what happened to the recurrence.

The whole point of the RNN was to carry information along the sequence, one step at a time, because that was the only way to get word one’s meaning to word forty. Attention now does that directly: computing $\alpha_{ij}$ needs no notion of sequential processing at all, just the two vectors being scored. The recurrence has quietly gone from being the mechanism to being the bottleneck it was hired to solve.

There is a second consequence, and it is the one that actually decided the field. Every $\alpha_{ij}$ for a given output step can be computed at the same time, because none of them depends on any other. The recurrence cannot: step $t$ needs step $t-1$ finished. One of those two things maps onto a GPU and the other does not, and that difference compounds every time somebody buys more hardware.

Three years later somebody deleted the RNN and kept only this. That is where the series goes next, and it is also where $Q$, $K$ and $V$ finally come from: not from the idea of attention, but from one specific way of packaging it once there is nothing else left in the model.