Do we really need keys, queries, values, and dot product attention?
Popular Internet analogies often explain attention by saying, “You need keys and queries for tokens to talk to each other.” But why? While there are many helpful analogies for how these components work, there is far less material on why they are necessary. Are there viable alternatives, or are these abstract concepts inevitable?
This might seem like a silly question, given the overwhelming success of the Transformer architecture by 2026. However, if history is any guide, Transformers will eventually be superseded by something more efficient or capable. The deeper we understand why they work, the faster we can innovate beyond them.
Remarkably, we can demonstrate that the general shape of the Transformer is difficult to avoid! Starting from a few key design pressures, the alphabet soup of matrices begins to appear far less arbitrary. Without relying on “token asks a question” analogies, we will see that queries emerge from a symmetry problem, while values and attention heads arise when we replace an unwieldy dynamic weight matrix with a compact set of reusable transformations.
As we reconcile our toy model with the original autoregressive Transformer, we will uncover one final fascinating connection: the MLP—the often-overlooked feedforward block—can be interpreted as its own key-value store.
Ready to invent the Transformer from scratch? First, we need to travel back to 2014 and ask why recurrent neural networks fell short.
Why Fixed Memory Fails
Standard recurrent neural networks (RNNs) have several limitations. Some, like the “vanishing gradient” problem, were addressed by the highly successful LSTM (Long Short-Term Memory) network by Hochreiter and Schmidhuber [1], a more advanced RNN variant. Yet, one core issue persists across all RNN flavors: they “squish” past inputs into a fixed-size memory.
To understand why, recall that an RNN maintains a memory state that is updated by the current input and the previous state. This temporal connection is the essence of “recurrence.” Consider a two-layer RNN unrolled over time:
Here, inputs are green, outputs are blue, and the two intermediate layers (computing standard neural network activations) are gray. All components are vectors, and the directed arrows represent matrix weights that multiply their inputs. To see the network at a single point in time, simply look at one vertical slice; in that slice, two fixed gray units collectively represent the “memory” at that moment.
Let’s use a crude analogy to illustrate why this “fixed memory” is problematic. Suppose you encode the sentence “I have five dollars” into the RNN’s memory, represented by the gray rectangle below:
Now, extend the sentence to “I have five dollars and forty cents in my pocket”:
Yikes. After adding more information into a finite memory, there is greater “competition” for space, and some information gets overwritten. This becomes disastrous when you need to recall specific facts or follow detailed instructions.
You might wonder, “Why not use dynamic memory that grows with the input sequence?”
Great intuition! This is exactly the direction we’ll explore next.