Who the menu was built for

In the first lesson, the token menu was built by counting. A program scans a huge pile of text and gives a piece of its own to whatever shows up most often.

That pile has mostly been English. So English words tend to earn whole pieces. Words in languages that were rare in the pile get chopped into many more.

Picture a bakery that cuts its dough into the shapes its regular customers order. The regulars get their loaf in one piece. Everyone else gets theirs assembled from scraps, and pays by the piece.

Measuring the gap

In 2023, Petrov and colleagues ran the same text, in about 200 languages, through many different tokenizers. Between languages they found gaps of up to 15 times. With the tokenizer then used by ChatGPT and GPT-4, Amharic took about 8 times as many tokens as English, and Burmese, Santali and Shan 12 to 15 times.

Because tokens are the unit everything is counted in, that gap has real effects for those speakers:

• Each message costs more when the service charges per token.
• Less of their text fits in the model’s window at once.
• The model has more pieces to work through, so it is slower.

Two people can ask a model the same question with the same meaning. If one asks in Amharic, that message is, in the model’s eyes, several times longer.

Hello, AI calls this a quiet unfairness. Nobody planned it. It falls out of counting.

Bigger menus help

One fix is a bigger menu, with room for more whole pieces in more languages.

Meta’s Llama 2 had a vocabulary of 32,000 tokens. Llama 3 has 128,256, about four times as many. Meta reported that English went from about 3.2 to 3.9 characters per token, so the same text needs about a fifth fewer tokens. Meta also added 28,000 pieces for other languages and said this made their text more compact too.

Hello, AI notes that today’s menus often run to around 200,000 pieces, grown largely to give other languages and computer code more whole pieces. The bills have moved much closer together.

The cause has not gone away. Frequency still decides who gets a whole piece. Bigger menus just give more languages a seat at the table.

In one sentence: tokenizers favor the languages they saw most, so the same message can cost some speakers many times more.