1. 1. Reading a price sheet
  2. 2. Input, output, cached
  3. 3. Check yourself
  4. 4. One chip, one dinner plate
  5. 5. Where the power goes
  6. 6. Check yourself
  7. 7. Smaller models, same answer
  8. 8. Watch a model type
  9. 9. Final check

Lesson 8 of 9 · Part 3 · Play · 2 min Draft

Watch a model type

Watch the same answer arrive at four speeds, from human reading pace to a specialised chip, and see what sets the limit.

The Speed fun-facts page has a live exhibit just below its cards. A short answer streams onto the screen one token at a time, each token a colored chip of text. Below it are four buttons, from human reading pace up to a specialised chip.

Try these three, slowest first:

  • Human reading. About 5 tokens a second, roughly how fast you read silently.
  • Cloud model. About 80 tokens a second, typical for a large model in the cloud. Notice it is already far ahead of your eyes.
  • The fastest button. The answer lands almost at once.

Then read the cards above it. Look for the one that explains the speed limit: each new token means reading all the weights from memory. Next, find the card where 4-bit weights raise that limit four times. Finish with the quiz card about serving many chats at once. It shows how one chip keeps many people’s replies flowing at once.

Watch a model type →

Quick check

One question to lock it in. It doesn’t count against you.

A server answers 64 conversations in one batch. What happens to each user's speed?

Sources