Part 1 · Cause (Mechanism)
Why attention disperses as tokens grow
Transformer attention computes token-to-token relations at N² cost. See how attention dilutes as the token count grows.
Note: this visualization's interface is in Korean.
Read the full notes as text
Every time an LLM decides on the next output token, it compares every token in the context against every other token. The context is everything so far: the tokens you typed in plus the tokens the model has produced. This pairwise comparison is called attention. With N tokens in the context, the number of pairs to compare grows to roughly N², so adding one more token does not cost “one more word”. It costs N new relations the model must track.
Tokenization: from sentence to integer IDs
Before a model can process natural language, the text is tokenized into fragments and each fragment is mapped to an integer ID from the vocabulary. From that point on the model computes relations between numbers, not words. With 8 tokens, 28 undirected relations are drawn (8×7/2), and since attention counts each direction separately and includes self-attention, the real number of comparisons is 64 (8×8).
Why N²
Every token asks every other token: “how much attention should I pay to you?” Two properties define this comparison.
- It is directional: A→B and B→A carry different values (the attention matrix is asymmetric).
- It includes self: each token also compares against itself (A→A).
With N tokens that makes N × N = N² questions: 100 for 10 tokens, 10,000 for 100 tokens, a million for 1,000 tokens.
Why attention ‘disperses’
Think of the model as having 100 points of attention to hand out for each output token. Those points are split across every token in the context: about 10 points each at 10 tokens, 0.1 points each at 1,000, 0.001 points each at 100,000. The same important word gets buried among its neighbors as the context grows. A token being present in the context does not mean the model sees it clearly.
| Tokens | Relations (N²) | Feels like |
|---|---|---|
| 10 | 100 | Sharp · 3 people in a room |
| 100 | 10,000 | Strained · 100 people in a hall |
| 1,000 | 1,000,000 | Dispersed · 1,000 people in a concert hall |
The truth about 1M windows
Models with 1M-token context windows exist now, but that does not mean the model’s working memory grew by the same amount. The region the model sees clearly, the Smart Zone, still sits at roughly 100k tokens. What grew is the Dumb Zone behind it, where attention thins out and information gets buried. A bigger window means more slots to fill, not a wider field of clear vision.
The football league analogy: adding a token = adding a team
A league with 4 teams needs 6 round-robin matches (4×3/2), 8 teams need 28 (8×7/2), and 100 teams need about 5,000(100×99/2 = 4,950). Adding a team is not “one more name on the roster”; the new team must play every existing team. Adding a token to the context has a similar ripple effect in attention.
Mind the numbers: a league pairs each two teams once, so matches grow as N(N-1)/2 (10 teams = 45 matches). Attention counts both directions and includes self, so it grows as N² (10 tokens = 100 comparisons). Both explode quadratically as N grows.
Bottom line: the cost of one more token is not “one more word to remember” but “N more relations the model must track”. The more tokens pile up, the harder it becomes for the model to decide where to focus.
Key takeaway
A full context window is a warning sign, not a badge. When the unit of work changes, clear the context with /clear.