Pieces from a fixed menu

When you type a sentence to a chatbot, the model never sees your words. Before anything else happens, the sentence is cut into standard pieces called tokens.

Think of a box of Lego bricks. Common shapes get a brick of their own. Anything unusual gets built from smaller bricks snapped together.

A model’s tokens work the same way. Each model comes with a fixed menu of pieces, chosen before it was trained. A very common word usually gets a single piece, with the space in front of it attached. A long rare word, a new name or a bit of slang may break into several pieces.

Nothing is ever off the menu. Rare things just cost more pieces.

Why not letters, or whole words

Letters alone would work, but every sentence would become very long. The model would spend its effort relearning how letters make words.

Whole words fail the other way. There are far too many of them, plus names, typos and new slang. No fixed menu could hold them all, and anything missing would be unreadable.

Tokens sit in between. Frequent chunks of text get their own entry. Everything else is assembled from smaller ones.

The menu is built by counting. A program scans a huge pile of text and keeps merging the pieces that most often sit side by side, until the menu is full. Frequency decides what earns a brick. Nobody asks a linguist, so the cuts can fall in places no dictionary would choose.

Numbers in, letters gone

Each token on the menu has an ID number, like a locker number at the gym. By the time your sentence reaches the model, it has become a row of these numbers. The letters have disappeared.

This explains an old chatbot stumble. Asked how many R’s are in “strawberry”, many models used to get it wrong. The question was about letters. The model only ever saw locker numbers.

Here is a handy rule of thumb from OpenAI. In ordinary English, one token is about four characters, or about three quarters of a word. So 1,000 tokens is roughly 750 words, about a page and a half. Other companies’ menus cut text differently, so the same words can count as more or fewer tokens. Computer code usually needs more tokens than ordinary prose.

Tokens are the unit the machine counts in. Prices, speed and how much text fits at once are all measured in them, as the next lessons show.

In one sentence: a model reads text as pieces from a fixed menu, and common words cost fewer pieces than rare ones.