Picture the chatbot as a sealed black box. You type "Let's plan Mom's birthday dinner" into one side, and a helpful reply comes out the other. In the last chapter you learned what's inside that box: billions of tuned knobs doing arithmetic. And that raises an awkward question. Arithmetic works on numbers. You typed words. Somewhere just inside the wall of that box, your sentence has to become numbers — without losing its meaning on the way. This chapter is about that doorway.

It happens in two moves. First, the sentence gets chopped into standard pieces called tokens. Second, each piece gets converted into a list of numbers — coordinates on a kind of giant map of meaning. Both moves are simple. Both have consequences you'll see every time you use a chatbot.

Move one: chop the sentence into tokens

A token is a chunk of text — sometimes a whole word, sometimes a piece of one — drawn from a fixed menu the model was built with. The model never works with your sentence directly; it works with the sequence of menu items your sentence maps onto. Here is roughly how our running example gets chopped:

Notice what just happened. Every token traded its letters for an ID number — a meaningless serial number, like a locker number. Common words like birthday and dinner are frequent enough in English that they earned a single slot on the menu. Rare words didn't: a name like Anastasiya might get chopped into An + ast + asi + ya, four tokens for one word. That's why chatbots sometimes stumble on unusual names, made-up words, or precise spelling — they're juggling fragments, not letters.

How big is the menu? A model's vocabulary — the full list of tokens it knows — is fixed when the model is built. GPT-2 and GPT-3 famously used exactly 50,257 entries, and we'll keep using that generation's numbers as our worked example throughout Part II. Today's menus are bigger: current OpenAI models use a roughly 200,000-token menu, and other frontier families run between 100,000 and 260,000 — mostly, as we'll see in a moment, to give languages other than English (and code) more whole bricks. Either way, the principle is the same: a few tens or hundreds of thousands of pieces are enough to spell any text in any language, the way a box of Lego bricks can build any shape — common things get a dedicated brick, everything else gets assembled from smaller ones.

Why chop at all — why not just use letters, or whole words? Both extremes fail. Letters alone would work (only ~100 symbols needed!) but make every sentence brutally long: our seven-token example would become thirty-plus pieces, and the model would spend all its effort re-learning that b-i-r-t-h-d-a-y spells a thing. Whole words fail the other way: English alone has hundreds of thousands of them, plus names, typos, and new slang — no fixed menu could hold every word the model might meet, and anything off-menu would be unreadable. Tokens are the compromise: frequent strings get their own entry, and any text — even "Anastasiya" or "yeetified" — can still be assembled from smaller pieces. Nothing is ever off-menu; rare things just cost more pieces. The menu itself is chosen before training by a simple greedy recipe: scan a huge pile of text, repeatedly merge the most frequent pair of adjacent pieces, stop when the menu is full — 50,257 slots for GPT-3, around 200,000 for today's models. Frequency in that pile decides what earns a brick, which is why English words fare better than, say, Thai ones — a quiet unfairness we'll revisit when we talk about model limits.

Because the menu was built by counting frequencies — not by consulting a linguist — the splits it produces are full of small surprises. A word you'd call "one unit" may be three tokens; a boundary may fall in a place no dictionary would put it. Here are six strings and what GPT-4's tokenizer actually does with them:

You type…	How it splits	Tokens	Why
birthday	birthday	1	Extremely common in the training pile, so it earned its own dedicated brick.
unbirthday	un + birthday	2	Too rare to earn a slot — but the greedy recipe reuses pieces it already has.
antidisestablishmentarianism	ant + idis + establish + ment + arian + ism	6	Long rare words become a chain of frequent fragments. The model juggles six lockers for one word.
ChatGPT	Chat + G + PT	3	Older tokenizers were frozen before the word existed. Newer ones, trained on text that mentions it constantly, give it fewer pieces — the menu is a snapshot of its era.
2026	202 + 6	2	This tokenizer chops numbers into chunks of up to three digits, so a year gets sliced mid-way — one reason models find digit-by-digit arithmetic awkward.
🎂	3 raw byte pieces	3	An emoji is stored as bytes, and rare bytes get no dedicated brick — so one visible symbol costs three tokens.

Notice the pattern: the boundaries are statistical, not linguistic. The splitter never asks "where does the prefix end?" — it asks "which chunks were frequent in my training pile?" When those two questions agree (as in un + birthday), splits look sensible; when they don't (as in arian + ism, or a year sliced into 202 + 6), you get boundaries no grammar book would draw.

That "quiet unfairness" from a moment ago is easy to see once you count. Take one sentence — "Let's plan a birthday dinner for Mom at a cozy Thai restaurant" — and say it four ways:

English~13

Spanish~30

Chinese~47

Python code~26

Move two: give each token a place on a map

Locker numbers have a problem: they're arbitrary. Token 8073 (dinner) is not "close to" token 8074 in any meaningful way, any more than locker 8073 at the gym holds gear similar to locker 8074. If the model worked with raw IDs, it would know nothing about meaning.

So there's a second conversion. Each token ID gets swapped for a word vector — a list of numbers that works like coordinates on a map. (A "vector" is nothing scarier than that: a list of numbers, like the pair of numbers that pins a spot on a city map.) The magic is in how the map is laid out: during training, the model arranges the map so that tokens used in similar ways end up near each other. Here's a toy two-number-per-word map for our dinner problem:

How does the model know where to put each word? It was never told. During training (the knob-tuning from Chapter 2), the map positions are knobs. Words that show up in interchangeable contexts — "we ordered pizza" / "we ordered sushi" — get nudged together, millions of nudges at a time, until the geography settles into something that mirrors how people actually use language.

Our toy map uses 2 numbers per word. Real models use thousands — GPT-3 used 12,288 numbers per token. You can't picture a 12,288-dimensional map (nobody can), but you don't need to. Think of the extra dimensions as scratch space for notes about meaning: a couple of dimensions' worth of room for "how food-related is this?", some for "how formal?", "singular or plural?", "positive or negative?", "French or English?" — thousands of tiny sliders, each capturing some faint shade of how the word behaves.

If maps feel too geometric, try a sommelier's trick instead. A wine-flavor wheel scores every bottle on dozens of axes — fruitiness, oakiness, acidity, tannin, minerality — and two wines with those scores nearly matching will taste like siblings even if their names share no letters: a Barolo and a Bordeaux can land side by side on the wheel while "Barolo" and "Bordeaux" only coincidentally both start with B. Word vectors are that wheel with thousands of axes: doctor and physician get near-identical scores and become neighbors, letters be damned. One place the analogy breaks: a sommelier can name each axis ("this one is oakiness"), while a word vector's axes were invented by training and mostly have no nameable flavor at all.

The party trick: arithmetic on meaning

Here's the payoff for going through all this trouble. Once words are coordinates, you can do arithmetic on meaning — and it works disturbingly well. The most famous example, discovered in real word-vector systems around 2013:

king − man + woman ≈ queen

On a map, subtracting one point from another gives you a direction — the arrow you'd walk to get from one to the other. Let's redo the trick with tiny made-up coordinates. Say man lives at (2, 1), woman at (6, 1), king at (2, 8), and queen at (6, 8). Then:

1. Subtract: king − man = (2, 8) − (2, 1) = (0, 7). That arrow — "go up 7" — is the royalty direction: what you add to a person to make them a monarch.
2. Add: take woman at (6, 1) and walk that same arrow: (6, 1) + (0, 7) = (6, 8).
3. Look up what lives there: (6, 8) is exactly where queen sits.

Nobody programmed "royalty" into the system. The direction emerged from nudging words around until similar contexts landed nearby. That's the deep lesson of this chapter: turn language into geometry, and a machine that only does arithmetic can start doing things that look like understanding.

The crack in the map: one word, one spot

But the map we've built has a flaw, and it's fatal. Every token gets exactly one spot — the same coordinates every time, no matter the sentence around it. Now consider:

• "We sat on the river bank." — mud, water, willows.
• "I need to stop by the bank." — marble, tellers, money.

One token, two utterly different meanings. Where should bank's single dot go — near river or near money? Any fixed choice is wrong half the time. Park it in the middle and it's wrong all the time, meaning something like "riverside ATM".

It bites our dinner plan too. Suppose your brother texts: "Make Mom's favorite dish!" Does dish mean her favorite food (pad thai) or her favorite piece of tableware (the blue ceramic one from Lisbon)? You resolve that instantly from context — obviously he means food, because you're planning a dinner. A static map can't. The dot for dish sits wherever it sits, deaf to every other word in the sentence.

What's missing is a mechanism that lets each word's coordinates shift depending on its neighbors — so that bank slides toward the river when river is nearby, and dish slides toward the food neighborhood when the sentence is about dinner. That mechanism exists. It's called attention, it's the single most important idea in modern AI, and it's the heart of the next chapter.

Where to go deeper

1. Easy: "Large language models, explained with a minimum of math and jargon" by Timothy B. Lee & Sean Trott — its opening section on word vectors covers this chapter's ground with more examples, zero equations.
2. Medium: "The Illustrated Word2Vec" by Jay Alammar — a gorgeous visual walkthrough of how word-vector maps are actually trained, including the king/queen arithmetic on real data.
3. Deep: 3Blue1Brown, "But what is a GPT?" — the first video in Grant Sanderson's transformer series; the segment on embeddings shows real high-dimensional word vectors and their directions, animated. Watching it after this chapter is the perfect bridge to Chapter 4.