Facts · Models · 7 cards
From 8 billion to a trillion knobs, in bytes you can picture
How big model files are
These facts come up in the course How AI Reads, where each one is explained step by step.
1
117 M
GPT-1 (2018) had 117 million parameters, the learned numbers inside a model. Stored at full precision it fits in 470 MB. Big phone games often take several gigabytes.
117 million parameters x 4 bytes each (fp32, full precision) = about 470 MB. The 117 million figure comes from the GPT-2 paper, whose smallest model matches the original GPT.
2
140 GB
File size is simply parameters times bytes per parameter. In bf16, the usual format for released weights, each parameter takes 2 bytes, so a 70B model needs about 140 GB.
70 billion parameters x 2 bytes (bf16) = 140 GB. Llama 3 70B is exactly 70.6 B, so 141 GB. Weights only, before any working memory.
Source: Llama Team, Meta, "The Llama 3 Herd of Models", 2024
3
562,500
One of the largest openly released dense models is 810 GB in bf16. Copied onto 1.44 MB floppy disks, the stack would stand about 1.9 km tall.
405 billion parameters x 2 bytes = 810 GB. 810 GB / 1.44 MB = 562,500 disks. At 3.3 mm per disk, that is about 1.9 km.
Source: Llama Team, Meta, "The Llama 3 Herd of Models", 2024
4
~70%
About 70 percent of Llama 3 8B is feed-forward (MLP) layers, which research suggests hold much of a model's stored knowledge. Attention, the famous part, is only about 17 percent.
Llama 3 8B: MLP per layer = 3 x 4,096 x 14,336 = 176 M; x 32 layers = 5.64 B of 8.03 B total. Attention is 42 M x 32 = 1.34 B.
Source: Llama Team, Meta, "The Llama 3 Herd of Models", 2024
5
Wait, really?
Only 37 billion, about 1 in 18.
It is a mixture of experts: a router sends each token through 8 of 256 small expert networks plus one shared one. But all 671 billion must stay in memory.
6
Moonshot's openly released mixture of experts has 384 experts, 8 active per token. In its native fp8 format (1 byte per parameter) it is about a terabyte on disk.
7
Guess first
About 200 GB
Int4 is half a byte per parameter: 405 billion x 0.5 = about 203 GB, a quarter of the bf16 size.
Source: Llama Team, Meta, "The Llama 3 Herd of Models", 2024
Each parameter is one number. Pick a model size and how many bits store each number.
70B parameters × 2 bytes = 140 GB