Lesson 2 of 11 · Part 1 · Play · 3 min Draft
Split a sentence yourself
Pick a sentence and see the exact pieces a real tokenizer cuts it into, and which words stay whole.
Below this text is a real tokenizer: the same kind of program that cuts your text into pieces before a model reads it. We ran nine sentences through it so you can see exactly where the cuts fall.
Pick each sentence in turn and keep an eye on the count:
- Everyday English. Most common words come out as one piece each, with the space in front attached.
- Long and rare words. These break into several pieces.
- Numbers and dates. Digits often split into small groups, so numbers cost more than they look.
- The same greeting in five languages. Watch the count. Some languages written in their own script, such as Burmese, need far more pieces than English.
None of these splits follow grammar. They follow what was common in the text the menu was built from.
See what the model sees
Pick a sentence. Each coloured block is one token, exactly as GPT-4o's tokenizer (o200k_base) splits it.
The cat sat on the mat and watched the rain.
The·cat·sat·on·the·mat·and·watched·the·rain.
11 tokens for 44 characters and 10 words
Unbelievably, the antidisestablishmentarian pharmacist hesitated.
Unbelievably,·the·antidisestablishmentarian·pharmacist·hesitated.
14 tokens for 65 characters and 5 words
On 2026-10-20 it cost $1,234.56 for 3,000,000 tokens.
On·2026-10-20·it·cost·$1,234.56·for·3,000,000·tokens.
25 tokens for 53 characters and 8 words
for i in range(10): print(i * i)
for·i·in·range(10):·print(i·*·i)
12 tokens for 32 characters and 7 words
Hello, how are you today?
Hello,·how·are·you·today?
7 tokens for 25 characters and 5 words
Hola, ¿cómo estás hoy?
Hola,·¿cómo·estás·hoy?
7 tokens for 22 characters and 4 words
你好，你今天怎么样？
你好，你今天怎么样？
5 tokens for 10 characters
नमस्ते, आज आप कैसे हैं?
नमस्ते,·आज·आप·कैसे·हैं?
10 tokens for 23 characters and 5 words
မင်္ဂလာပါ၊ ဒီနေ့ နေကောင်းလား။
မင်္ဂလာပါ၊·ဒီနေ့·နေကောင်းလား။
16 tokens for 29 characters
A · marks a space. Spaces usually ride along at the front of the next word.
Quick check
One question to lock it in. It doesn’t count against you.