Facts · Inference · 7 cards
Why a model types faster than you read, and what slows it down
Tokens per second, from reading to wafer-scale chips
These facts come up in the course What AI Costs, where each one is explained step by step.
1
~80 tok/s
A frontier model through a cloud API streams 50 to 150 tokens (word pieces) a second; about 80 is typical. A person reading silently manages about 5.
Adults read about 240 words a minute. At roughly 0.75 words per token that is 320 tokens a minute, or about 5 a second. 80 / 5 = 16.
Source: Artificial Analysis, 2026
2
2,100 tok/s
Cerebras served Llama 3.1 70B at about 2,100 tokens a second in late 2024 by keeping the weights in on-chip memory, far faster than a GPU's.
A 100,000-word novel at 0.75 words per token is about 133,000 tokens. At 2,100 tokens a second that is roughly 63 seconds.
Source: Cerebras, 2024
3
209 tok/s
Every new token means reading all the model's weights once. For an 8B model on one H100, that caps a single conversation near 209 tokens a second.
8B parameters x 2 bytes (bf16) = 16 GB. The H100 reads memory at 3,350 GB/s. 3,350 / 16 = about 209 reads, so 209 tokens a second. Real systems reach 60 to 80 percent.
Source: NVIDIA H100 datasheet
4
4-bit weights are a quarter of the bytes, so the same memory pipe delivers four times as many tokens. The cost is a little accuracy.
Source: NVIDIA H100 datasheet
5
~25 s
Before answering, the model must read the whole prompt. A 405B model given a 100,000-token prompt needs about 25 seconds on eight H100s. An 8B model: half a second.
2 x 405B parameters x 100,000 tokens = 8.1 x 10^16 operations. Eight H100s at 40 percent of their 989 TFLOPS peak give about 3.2 PFLOPS. 8.1 x 10^16 / 3.2 x 10^15 = about 25 s.
Source: NVIDIA H100 datasheet
6
Wait, really?
No. The output is identical to the slow path.
The big model verifies several guessed tokens in one pass, which costs about the same as writing one. Accepted guesses are free; rejected ones are simply redone.
Source: Leviathan et al., 2023
7
Guess first
It stays nearly the same
The weights are read once per step for everyone, so each conversation gets a token from the same read. Total output climbs many times over, while each user's own speed falls only a little.
Source: Yu et al., Orca, OSDI 2022
A model writes one token at a time. Pick a speed and watch the same 48-token answer arrive.
A model writes one token at a time. Each new piece depends on every piece before it, so the answer arrives word by word. Pick a speed and watch the same short answer appear at human, laptop, cloud and specialised-chip pace.
Pick a speed to start.