Even after engineers mastered training neural weights with backpropagation, early language models suffered from a crippling flaw: digital amnesia over long text. Older architectures read sentences strictly one word at a time from left to right. By the time the network reached the fiftieth word of a paragraph, the opening words had faded away. In 2017, a team of researchers, most of them at Google, broke that bottleneck with a landmark paper titled “Attention Is All You Need,” introducing the Transformer architecture that underlies today’s large language models.
1. Parallel Reading: Seeing the Whole Passage at Once
Instead of reading word by word like a slow reader following a finger, a Transformer takes in a whole passage at once, up to a size limit called its context window. To keep track of word order without reading in sequence, engineers add a mathematical code to every word called a positional encoding. This lets the network process thousands of words in parallel across GPU clusters while still knowing where each word sits.
Illustration only: arc thickness shows the idea of strong versus weak attention. It is not measured from a real model.
2. Self-Attention: The Query, Key and Value Engine
The beating heart of the Transformer is a mechanism called self-attention. Consider the sentence: “The trophy didn’t fit in the suitcase because it was too big.” A human knows instantly that “it” is the trophy. If the sentence ended “because it was too small,” “it” would be the suitcase. Linguists call such pairs Winograd schemas, and they are hard for software because only meaning, not grammar, settles the answer. To copy this contextual intuition, every word produces three lists of numbers inside the network:
Query (Q), the search question: each word asks the other words, “Which of you help define my meaning?”
Key (K), the identity tag: every word holds up a Key advertising what it has to offer.
Value (V), the actual content: when a Query matches a Key strongly, the word takes in that word’s Value, blending the meanings together.
These labels are an analogy. In a real network, Q, K and V are learned numbers produced by multiplying each word’s representation by weight matrices that training adjusts (Part 3). The original paper writes the whole step as one line:
Attention(Q, K, V) = softmax( Q·Kᵀ / √d ) · V
Worked example: turning scores into attention weights
Suppose the word “it” scores 2.0 against “trophy,” 1.0 against “suitcase” and 0.1 against “too big.” The softmax step converts scores into weights that add up to 100%:
e2.0 = 7.39, e1.0 = 2.72, e0.1 = 1.11. Their sum is 11.21.
Weights: 7.39 ÷ 11.21 ≈ 66%, 2.72 ÷ 11.21 ≈ 24%, 1.11 ÷ 11.21 ≈ 10%.
“It” then takes 66% of the trophy’s Value, 24% of the suitcase’s and 10% of the third word’s. The scores here are made up to show the arithmetic. The √d in the formula only keeps the scores from growing too large, which would make training unstable.
3. Multi-Head Attention: Watching Several Angles at Once
One round of attention can only focus on one pattern of relationships. So the Transformer runs several attention calculations side by side, called heads, each with its own Q, K and V weights, and then combines their results. The original base model used 8 heads in every layer.
Nobody assigns the heads their jobs. Researchers who later examined trained models found that some heads settle into recognizable habits:
| Attention Head Role | What It Tends to Track | What Researchers Found |
|---|---|---|
| Positional heads | The word just before or after the current one. | Found in a translation model, where a few specialized heads did most of the work (Voita et al., 2019). |
| Syntactic heads | Grammar links, such as a verb and its object. | Some heads in BERT line up with grammatical relations (Clark et al., 2019). |
| Rare-word heads | Uncommon words that carry a lot of meaning. | Also reported by Voita et al. (2019). Many other heads could be removed with little loss. |
Treat these roles as tendencies, not guarantees. In large models, most heads do not fit a tidy label, and what each one contributes is still an active research question.
4. What Happens When You Scale Transformers to Trillions of Weights?
Because attention runs in parallel, Transformers fit GPUs well, and that made enormous scaling practical. The published milestones show how far it went:
- Original Transformer (2017): the base model had 6 layers and 8 attention heads, with about 65 million parameters.
- GPT-3 (2020): 175 billion parameters in 96 layers. At 16-bit precision, its weights alone take about 350 GB.
- Switch Transformer (2021): up to 1.6 trillion parameters, using a “mixture of experts” design in which each word is routed to only a small part of the network, so most of the weights sit idle for any one word.
Scaling laws. Experiments by Kaplan et al. (2020) showed that a language model’s error falls in a smooth, predictable pattern as model size, training data and computing power grow. That predictability is a major reason companies kept spending on larger systems.
Bigger is not the whole answer. In 2022, the Chinchilla study found that many large models were under-trained. A 70-billion-parameter model trained on about 1.4 trillion words beat much larger models trained on less data. For a given computing budget, size and data have to grow together.
New abilities, or a measuring trick? Some skills seem to appear suddenly once models pass a certain size. Other researchers argue that this sudden jump can come from how the test is scored, not from the model (Schaeffer et al., 2023). The question is still debated.
The physical cost. Every one of those weights has to be stored and moved between memory and processors. That is why scale becomes a hardware and power problem. For the engineering behind it, see our data center series on AI memory and storage architecture.
Key takeaways
- A Transformer reads a whole passage at once and uses positional codes to keep word order.
- Self-attention lets every word weigh every other word using learned Query, Key and Value numbers.
- Multi-head attention runs several attention patterns in parallel, and their roles are learned, not assigned.
- Scaling laws made bigger models predictable, but training data matters as much as size.
- Very large models are a memory and power problem as much as a math problem.
References
- Vaswani, A. et al. (2017). “Attention Is All You Need.” NeurIPS 30.
- Levesque, H., Davis, E. and Morgenstern, L. (2012). “The Winograd Schema Challenge.” KR 2012.
- Voita, E. et al. (2019). “Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned.” ACL.
- Clark, K. et al. (2019). “What Does BERT Look At? An Analysis of BERT’s Attention.” BlackboxNLP.
- Brown, T. B. et al. (2020). “Language Models are Few-Shot Learners.” NeurIPS 33.
- Kaplan, J. et al. (2020). “Scaling Laws for Neural Language Models.” arXiv:2001.08361.
- Fedus, W., Zoph, B. and Shazeer, N. (2022). “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” JMLR 23.
- Hoffmann, J. et al. (2022). “Training Compute-Optimal Large Language Models.” NeurIPS 35.
- Schaeffer, R., Miranda, B. and Koyejo, S. (2023). “Are Emergent Abilities of Large Language Models a Mirage?” NeurIPS 36.
Educational content. Examples are simplified to explain the ideas, and model sizes come from the published papers listed above.
- Part 1: Inside the Synthetic Neuron — How AI Brains Turn Random Noise into Digital Memory
- Part 2: Deep Neural Layers — How Millions of Artificial Weights Form a Thinking Network
- Part 3: Backpropagation & Loss — How an AI Brain Corrects Its Own Mistakes During Training
- Part 4: The Attention Revolution — Inside the Transformer Architecture That Gave AI Context
- Part 5: Parameter Scaling Laws — Why Modern AI Brains Grew from Millions to Trillions of Weights
- Part 6: Dense vs. Mixture of Experts (MoE) — Inside the Architectural Divide of US and Chinese AI
- Part 7: Reinforcement Learning & Reward Models — How AI Brains Learn from Reward and Punishment
- Part 8: Test-Time Compute & Reasoning Drafts — How Modern AI Brains Think Before Speaking
- Part 9: AI Data Centers & Hardware — Why Massive Brains Require H100 Clusters and High VRAM
- Part 10 (Series Finale): The Architecture of AGI — Combining Memory, Reasoning, and Scale Toward General Intelligence
