Transformers power almost every modern large language model, yet the core idea is surprisingly approachable. This post walks through the pieces without the heavy math.
From text to tokens
Models don’t read words — they read tokens. A tokenizer splits text into chunks
(whole words, word fragments, or single characters) and maps each to an integer ID.
The phrase token hungry might become ["token", " hungry"], then [3092, 27089].
Each token ID is looked up in an embedding table, turning it into a vector — a list of numbers that positions the token in a high-dimensional space where similar meanings sit near each other.
Attention: the key idea
The breakthrough of the transformer is self-attention. For every token, the model asks: which other tokens should I pay attention to in order to understand this one?
Each token produces three vectors:
- Query — what am I looking for?
- Key — what do I offer?
- Value — what information do I carry?
A token’s query is compared against every other token’s key to produce attention weights. Those weights then blend the value vectors together. In a sentence like “the model read the token because it was hungry,” attention is what lets it connect back to the model.
Stacking layers
A single attention step is useful; stacking dozens of them is powerful. Each transformer block combines:
- Multi-head attention (several attention patterns computed in parallel)
- A feed-forward network applied to each position
- Residual connections and normalization to keep training stable
Deeper stacks capture increasingly abstract structure — syntax in early layers, meaning and reasoning patterns in later ones.
Why they scale
Unlike older recurrent models, attention processes every token in parallel rather than one step at a time. That parallelism maps beautifully onto modern GPUs, which is a big reason transformers scaled from millions to trillions of parameters so quickly.
Takeaways
- Text becomes tokens, tokens become vectors.
- Attention lets each token draw context from every other token.
- Depth adds abstraction; parallelism enables scale.
That’s the whole machine, minus the math. Everything else — bigger datasets, longer context windows, cleverer training — is refinement on top of this foundation.
