Facts · Models · 7 cards
The pieces models read, and how many fit in one go
Vocabulary size: Llama 2 vs Llama 3
These facts come up in the course How AI Reads, where each one is explained step by step.
1
Wait, really?
The word "distribute". Other ghost tokens drew insults or unrelated words.
Odd strings such as Reddit usernames became single tokens in the GPT-2 and GPT-3 tokenizer, probably because the model rarely saw them in training. Many tokenizers have ghost tokens like this.
2
~0.75 words
Models read tokens, word fragments, not letters or words. Common words are one token with their leading space attached; rare words break into pieces.
For OpenAI's tokenizers, English prose runs about 4 characters, or 0.75 words, per token. So 1,000 tokens is about 750 words, a page and a half. Other companies' tokenizers can count the same text quite differently.
Source: OpenAI Help Center, What are tokens and how to count them
3
8–12×
Tokenizers are trained mostly on English, so rarer scripts split into many more pieces. Those speakers pay more per message and fit less in the window.
Petrov et al. ran the same text in about 200 languages through many tokenizers and found gaps of up to 15x. With the tokenizer used by ChatGPT and GPT-4 in 2023, Amharic took about 8 times as many tokens as English and Burmese about 12 times.
Source: Petrov et al., NeurIPS 2023
4
The bigger vocabulary raised English from 3.17 to 3.94 characters per token, so the same text needs about a fifth fewer tokens. Meta also added 28,000 tokens for other languages, which it says made their text more compact too.
5
1.7 × 10¹⁰
Attention, how a model relates words, compares every token with every other. Four times the tokens means sixteen times the work.
128K is 131,072 tokens, and 131,072 x 131,072 is about 1.7 x 10^10 pairs. At 1 million tokens it is 10^12. FlashAttention avoids storing the full grid, but the arithmetic still has to be done.
Source: Dao et al., FlashAttention, 2022
6
$2–10
Context is billed per token per request, so a long document re-sent each turn is paid for each turn. Prompt caching makes repeat reads much cheaper.
On Anthropic's October 2026 prices, 1,000,000 input tokens cost $2 on Claude Sonnet 5.5, $4 on Opus 5.5 and $10 on Fable 5.1, with no extra long-context rate. Cached reads cost 5 percent of the input price on Sonnet and Opus 5.5.
Source: Anthropic pricing, October 2026
7
Guess first
In the middle
The "lost in the middle" study found models used facts at the start or end of a long prompt much better than facts placed in the middle.