How Attention Works in a Transformer

The model takes a piece of text and gives back a list of probabilities. The list has one probability for every word or symbol that could come next. The probabilities add up to 100%. Figure 1 shows the top of the list for both texts.

Without “Golden”, the probability spreads out over many options. With “Golden” in front of “Gate”, most of the probability goes to one word.

To write a longer text, the model makes the same kind of guess again for each new word.

The model works on numbers, so a text has to be turned into numbers before it goes in. First, the text is cut into pieces called tokens. Then each token is replaced by a list of numbers.

A token is a word or a piece of a word. Common words such as “Golden” and “Gate” are one token each. A rarer word is split into pieces. For example, “Gatekeeper” becomes “Gate” followed by “keeper”. Punctuation marks and line breaks are tokens too. GPT-2’s vocabulary is a fixed list of 50,257 tokens. We write that number as N. In our example, each word is exactly one token.

The model has a token table, also called an embedding table, with one row for each of the N tokens. Each row holds 768 numbers. A list of numbers like this is called a vector. The vector’s length is the model’s width. We write it as d, so in GPT-2 small, d is 768.

The row for “Gate” starts with −0.033, −0.348, 0.317, −0.164, −0.062, 0.079, −0.227, −0.096, and 760 more numbers follow. “Gate” starts from the same row in every text that goes into the model.

The token table holds no information about word order. “Gate” gets the same row whether it comes first or tenth. So the model has a second table, the position table. It has one row of d numbers for each place in a text: first, second, third and so on, up to C places. C is the context window, the most tokens the model takes in at once. For GPT-2 small, C is 1,024. In our example, “the” is first, “Golden” is second and “Gate” is third.

Each token’s row from the token table is added to the row for its place, number by number. The sum is the token’s starting vector. So the same word starts out with slightly different numbers in different places.

Next, the three starting vectors pass through GPT-2 small’s 12 layers, one after another. Every layer has the same two steps, attention and then feed-forward. Each layer has its own parameters. Each token’s vector runs straight through all 12 layers as one line, called its residual stream. The residual stream stays d numbers long the whole way. After the last layer, the output step turns the residual stream of “Gate” into the probabilities in Figure 1.

Figure 2 shows the three residual streams going through the 12 layers. Under each layer is the model’s best guess for the next word if we stopped after that layer.

Every layer has the same two steps: attention first, then feed-forward. All sharing of information between tokens happens in attention. Feed-forward then works on each token by itself, with everything attention has brought in. Each step produces an update: a list of d numbers added to each token’s residual stream. In the figures, += marks the spot where an update is added. Attention is split into 12 parts that run side by side, called heads.

A layer holds its parameters in six matrices. A matrix is a block of numbers laid out in rows and columns. The model does most of its arithmetic with two operations. A dot product, written ·, multiplies two vectors number by number and adds up the results. Matrix multiplication, written ×, takes the dot product of a vector with each column of a matrix. The result is a new vector, with one number for each column.

Figure 4 shows the six matrices and what each one does. Attention uses four of them. Three make each token’s query (Q), key (K) and value (V), and the fourth is the combining matrix. Feed-forward uses the other two, the up-projection and the down-projection.

Figure 5 shows how much space each part of GPT-2 small takes up in the downloaded file.

Attention adds content to each token’s residual stream from the earlier tokens and from the token itself.

A search sets how much comes from each token. Each head gives every token so far a share. These shares add up to 100%. Figure 6 shows the shares when “Gate” comes through layer 3.

Each head splits the 100% in its own way. In heads 3 and 5, “Golden” gets the largest share. In the other heads, “the” gets the largest share.

The tokens go through each layer one at a time: first “the”, then “Golden”, then “Gate”. Each token’s K and V go into the layer’s KV cache and stay there for every later token.

When a token comes through, its Q is compared with every K in the KV cache, its own included. The result of each comparison is a score. The scores are then turned into the shares, which add up to 100%. A bigger score gets a bigger share. Each V in the KV cache is multiplied by its share. Adding these weighted values gives the head’s result.

All 12 heads search at the same time, and each head gives a result of d/12 numbers. The 12 results are joined end to end into d numbers and multiplied by the combining matrix, as Figure 9 shows. The result, attention’s update, is added to the residual stream.

After attention, feed-forward works on each token’s residual stream by itself. It compares the residual stream with 4d patterns, the columns of the up-projection. GPT-2’s designers made feed-forward four times as wide as the model, as in the original transformer. Each comparison is a dot product, called a match. Each matching pattern adds a vector from its row in the down-projection. A stronger match adds more of that vector.

In layer 1, the best guess is “Gate” itself, both after attention and after feed-forward.

The word “Golden” changes the model’s guess about what comes after “Gate”. With “Golden” in the text, the top guess is “Bridge”, at 84.8%. Without “Golden”, the top guess is “of”, at 20.1%.

Information from “Golden” can reach “Gate” in one way, through attention. Figure 13 shows the share of “Golden” in every head of every layer.

Most of the darker squares sit in layers 1 to 5.

After layer 12, the output step takes the dot product of the residual stream of “Gate” with every row of the token table. This is the same table the tokens started from. The results become probabilities that add up to 100%.

One token is then picked at random, and a likely token gets picked more often. The pick works like a draw from a hat in which 84.8% of the tickets have “Bridge” on them. With the randomness turned off, the top token comes out every time. Chat apps run with some randomness turned on, so asking again can give a different answer. The same randomness can be steered to hide a watermark in the text, and Did AI Write This? shows how.

The new token joins the other three and goes through the 12 layers the same way. In every layer, its Q is compared with every K in that layer’s KV cache. The KV cache holds a K from each of “the”, “Golden” and “Gate”, and the new token’s K joins them. The K and V of each earlier token are already in the KV caches, so those tokens don’t go through the layers again.

We started with three words, “the Golden Gate”. Each one became a list of numbers. Attention brought information from the earlier words into the residual stream of “Gate”. Feed-forward then worked on that stream with its learned patterns. After 12 layers, the output step gave “Bridge” an 84.8% chance of coming next.

The transformer comes from a 2017 paper, Attention Is All You Need. With attention, every token reaches every earlier token in a single step. During training, the calculations for all the tokens in a text can run at the same time. Graphics chips run that kind of work fast, so researchers could train much larger models on much more text. Researchers have kept changing the design since then.

“Golden Gate” is a familiar phrase, so “Bridge” may seem an easy guess. In October 2026, as we write this, models with attention in every layer are settling open problems in mathematics, some of them 80 years old.

Attention began in 2014 with Dzmitry Bahdanau, a 23-year-old intern in Montreal. He took the idea from translation exercises in his school English lessons. For each word he wrote, he had glanced back over the original sentence.

• Every figure shows the model in use, with its parameters already set. Engineers call this use inference. We left out training, the process that sets the parameters.
• Each step reads a rescaled copy of the residual stream, and so does the output step. The rescaling keeps the numbers in a steady range. We left it out of the figures.
• Each head’s Q and K hold d/12 numbers, 64 in GPT-2 small, and attention divides every score by 8, the square root of 64. The division keeps the scores from growing with the length of Q and K. The output step’s scores are left as they are.
• In the figures, the tokens go through each layer one at a time. All the tokens can also go through a layer together, and the numbers come out the same.
• In normal use, the output step runs once, after layer 12. To show the guesses in Figures 2 and 12, we also ran it after each earlier layer.
• The rule in feed-forward is a smooth curve called GELU. Negative numbers come out of it a little below zero.
• For the drawing in 1.2’s intuition box, we took the token-table rows of “Bridge” and “of”, subtracted the average row from each, and drew the flat slice that holds both. The vector of “Gate” is drawn as it reaches the output step, after its last rescaling. Subtracting the average row changes no probability.
• The three larger GPT-2 sizes have more layers and wider vectors, and so do the transformers behind today’s chatbots. Their layers are built on the same plan.

If this felt too dry and you’d like an intuitive feel for attention, Who Killed Aldous Finch? explains the same ideas through a country house murder mystery. Each new witness reads every placard already pinned in the room and then pins up one more. At the end, each clue gets matched to a part of the transformer.

Two more pieces on this site are about how artificial minds learn. In Who Learns Faster, a Toddler or a Chatbot?, a child and Meta’s Llama 3.1 go side by side through four stages of learning. The model read trillions of words before it could hold a conversation. Inside one chat, it picks up a new skill in a minute.

An Artificial Mind of Your Own explains how an artificial mind is trained and how to run one on a laptop.