A full context window has a cost you never see. The Memory fun-facts page shows it.

Here is the idea to hold onto. As the model reads each token, it leaves itself notes about it, so later words can look back without redoing the work. Engineers call these notes the KV cache. They sit in the memory of the chips that run the model, and they grow with every token in the window.

As you read the cards, look for these:

• How much memory a single token leaves behind in Llama 3 70B, a large open model.
• What that adds up to for one 128,000-token chat. Compare it with the memory needed just to load the model.
• The card where a bigger model needs less chat memory than a smaller one. How a model is built changes the size of its notes.

Then take the quiz card and guess how many tokens fit on a nearly full chip.