Facts · Inference · 7 cards
Where the gigabytes go when a model remembers your chat
Chat memory for one Llama 3 70B conversation
These facts come up in the course How AI Reads, where each one is explained step by step.
1
42 GB
A single 128K-token conversation with Llama 3 70B carries 42 GB of KV cache (the model's notes on every past token). That is half an 80 GB H100 GPU.
320 KiB per token × 128,000 tokens ≈ 42 GB. At 8K tokens the same chat needs 2.6 GB; at 32K, 10.5 GB.
2
141 GB
Full-quality serving takes 2 bytes per parameter in bf16 (16-bit numbers). An 8B model is 16 GB; a 70B model is 141 GB; a 405B model is 810 GB.
Parameters × 2 bytes. About 70.6 billion × 2 ≈ 141 GB, more than one 80 GB GPU holds.
3
320 KiB
Every layer saves a key and a value for each token, so later words can look back without recomputing. In Llama 3 70B that totals 320 KiB per token.
2 (key and value) × 80 layers × 8 KV heads × 128 numbers per head × 2 bytes = 327,680 bytes = 320 KiB.
4
Wait, really?
Yes: 604 GB versus 66 GB at 128K tokens.
GPT-3 kept keys and values for all 96 heads. Llama 3.1 405B uses grouped-query attention, so its 128 query heads share just 8 KV heads. How a model is built matters as much as its size.
Source: Ainslie et al., GQA, 2023
5
DeepSeek-V3 uses latent attention: it squeezes each layer's keys and values into one 576-number vector per token, about 70 KB per token in all.
6
< 4%
Servers used to reserve room for each request's longest possible chat, mostly empty. vLLM's PagedAttention hands out cache in small pages on demand, like an operating system.
7
Guess first
About 25,000
About 8 GB remains after overhead. At 320 KiB per token that is roughly 25,000 tokens across all users. One long document fills it.