Keep the black box sealed one more time, but now ask a different question about it. Not "what does it do?" — you know that from Chapter 4: predict the next token, loop. Ask instead: where did the box come from? Because here's the puzzle. A next-token predictor should behave like a parrot with a library card — continuing whatever text you start. Yet when you type "Suggest a restaurant for Mom's birthday dinner," it doesn't continue your sentence. It answers you, politely, in a helpful-assistant voice, and stops when it's done. Nobody programmed that voice line by line. It was grown, in three distinct stages — and once you know the stages, a dozen quirks of chatbots suddenly make sense.

Stage 1 — Pretraining: read the internet, predict the next token

1. Gather a mountain of text. Web pages, books, code, articles, forums — cleaned and filtered into a diet of trillions of tokens. (Recall the scale: tens of thousands of times more words than a person hears in childhood.)
2. Play the prediction game, everywhere. Take a random snippet, hide the next token, let the model guess. "Happy birthday to ___" — the model guesses; the truth is revealed; every one of its billions of knobs gets nudged a hair toward whatever would have made the right answer more likely. This is exactly Chapter 2's tune-the-knobs loop, run on trillions of hidden-word guesses.
3. Repeat for months. Not on one computer — on tens of thousands of specialized chips (GPUs) running in parallel, around the clock, for months.

Step 1 hides a surprise worth drawing: most of the work of building a training diet is throwing text away. The raw web is a landfill — spam, duplicates, gibberish, boilerplate, the same press release ten thousand times — and model quality tracks diet quality (Chapter 2's "data is destiny"). So the pipeline is a funnel that discards far more than it keeps:

The result is called a base model: a machine that has gotten astonishingly good at one game — continue the text. To get good at that game across all text, it was forced to absorb spelling, grammar, facts, styles, some reasoning, many languages. All of Chapter 4's machinery — the meaning-map, attention, the knowledge in the feed-forward layers — is built during this stage.

But a base model is not a helpful assistant, and seeing the difference is the fastest way to understand everything that follows. Same input, two very different machines:

You type…	Base model (continues the text)	Assistant model (answers you)
Suggest a restaurant for Mom's birthday dinner.	"Suggest a venue for the after-party. Suggest a gift under $50. This worksheet is part of our Family Events planning series, page 3 of 12…"	"Happy planning! A few questions first: what cuisine does she love, and what's your budget? If she likes Thai, a quiet upscale Thai place is a great birthday pick — book a week ahead…"

The base model isn't broken — it's doing its job perfectly. Asked "what text plausibly follows a lone imperative sentence like that?", a decent guess is: more of the same worksheet. It read your words as a document to extend, not a request to fulfill. Nothing in next-token prediction says anyone is talking to it. (Both outputs here are illustrative, precomputed for this book — but this is faithfully what raw base models do.)

Stage 2 — Supervised fine-tuning: show it what an assistant sounds like

How do you turn a text-continuer into a conversation-holder? Not by reprogramming — by more training, on a very different diet. Supervised fine-tuning (SFT) means hiring people to write thousands of model conversations — a user message, then the ideal assistant reply — covering questions, coding help, refusals of harmful requests, admissions of uncertainty, and yes, event planning. Then the same knob-tuning runs again on this small, curated set: predict the next token, but now the text being predicted is exemplary assistant dialogue.

Thousands of examples can't teach new knowledge — that took trillions of tokens. What they teach is format and persona: when text is arranged as a conversation, the thing to predict after a user's question is a helpful answer that then stops. The base model's vast abilities get channeled into a character. Researchers sometimes put it this way: pretraining builds the capabilities, fine-tuning teaches the model which character to play.

Stage 3 — Preference training: reward the better answer

After SFT the model talks like an assistant, but "like an assistant" leaves a lot unpinned: it may ramble, guess confidently, or comply with sketchy requests. The problem is that "a good answer" is easy to recognize but hard to write rules for. Stage 3 exploits the recognizing:

1. Generate two answers to the same prompt — say, two dinner suggestions, one thoughtful, one generic.
2. A human (or, increasingly, another AI) picks the better one. Just a choice: A or B.
3. Turn those picks into a reward signal — a learned score for "answers people prefer" — and tune the knobs to chase it, nudging the model toward the kinds of answers that win comparisons and away from the kinds that lose.

The famous version is RLHF — reinforcement learning from human feedback ("reinforcement learning" = learning from reward signals rather than from example text). Newer variants like DPO (skipping the middleman score) and RLAIF (AI raters guided by a written constitution, as in Anthropic's approach) differ in plumbing, not in spirit: compare, prefer, reinforce. The target is often summarized as the three H's — helpful, harmless, honest. This stage is why chatbots hedge, add caveats, decline certain requests, and generally feel like a careful concierge rather than a wild text generator. (It's also, critics note, why they can feel a bit same-y and over-agreeable — you get what raters reward, including, sometimes, flattery.)

That last parenthesis has a famous real-world case study, dated and documented in the provider's own postmortem. In April 2025, OpenAI shipped an update to GPT-4o that folded a new signal into preference training: users' thumbs-up and thumbs-down clicks. Reasonable-sounding — who knows what users prefer better than users? But the new signal, OpenAI later wrote, "weakened the influence of our primary reward signal, which had been holding sycophancy in check." The model turned into a flatterer: validating doubts, fueling anger, cheering on impulsive decisions. The part that should stick with you is how it slipped through: the offline evaluations and A/B tests all looked good — people, in the moment, click thumbs-up on agreement — while expert testers' informal "something feels off" impressions were set aside. OpenAI rolled the update back within days and committed to treating model behavior, not just capability, as launch-blocking. It's the cleanest proof you'll find of this chapter's warning: "answers people prefer" is a blunt instrument, and a model optimized to please can be optimized away from being honest.

A human version of the whole arc: the base model is someone who has read the entire library — every book, twice — but has never once held a conversation; ask them a question and they recite the shelf your question reminded them of. SFT is etiquette school: a few weeks of "when someone asks, you answer, then stop talking." And RLHF is what turns a trained conversationalist into a good one — going to dinner parties and getting frank feedback afterward ("you rambled," "you should have said you didn't know") until the manners become instinct. The analogy's weak spot: a person at etiquette school understands why the rules exist; the model only learns which token patterns win, which is why its manners can be gamed in ways a socialized human's can't.

Side by side, the three stages look like this — read the last column twice, because what breaks when a stage is skipped is the best proof of what that stage actually does:

Stage	What the model practices	Data needed	Rough cost share	What you'd notice if it were skipped
1 · Pretraining	Predict the next token on ordinary text	Trillions of tokens (the filtered internet)	~95%+ of the bill; months	Everything. No language, no facts, no grammar — the model is an empty machine that emits noise.
2 · Supervised fine-tuning	Imitate exemplary assistant dialogue	Thousands to hundreds of thousands of written conversations	A small slice; days	Ask "Suggest a restaurant for Mom's birthday" and it continues your worksheet — "Suggest a venue for the after-party. Page 3 of 12…" — instead of answering you.
3 · Preference training	Produce answers humans pick as better	Hundreds of thousands to millions of A-or-B comparisons	A small slice; weeks	It answers, but badly: rambling, confidently wrong, over-compliant — happy to help with requests it should decline, in a voice no one polished.

The fourth stage: teaching the model to think before it speaks

Chapter 4 left you with the train-car problem: tokens can't be un-written, so a model that commits to a wrong turn in its first sentence elaborates the mistake instead of fixing it. Chapter 6 will hand you a partial home remedy — ask the model to "reason step by step first," because tokens are the model's only scratch paper: whatever it writes out becomes context its later tokens can attend to. Around 2024, the labs stopped treating that trick as a prompting tip and started training it in. The result is the "reasoning" or "thinking" model — the biggest change in how LLMs are built since chat itself, and the toggle now sitting in every major chat app.

Here's the recipe, and notice that you already know every ingredient. Take problems with checkable answers — math with a verifiable solution, code that either passes its tests or doesn't. Let the model write as much private scratch-work as it wants before its final answer. Then reward it, Chapter 2's dog-and-treats style, when the final answer checks out. No human writes out the reasoning to imitate; no rater judges which answer "sounds better." The reward is being right, machine-checked — which neatly sidesteps the flattery trap of stage 3, because a math checker cannot be charmed.

What emerges from that training loop surprised even its builders. OpenAI's o1 (late 2024) was the first mainstream model of this kind, "trained with large-scale reinforcement learning to reason using chain of thought." A few weeks later, DeepSeek's R1 — whose training write-up was eventually published in the journal Nature, a first for a frontier LLM — showed that reasoning can be incentivized "through pure reinforcement learning," with no human-labeled reasoning examples at all: behaviors like self-reflection and double-checking emerged spontaneously, because pausing to verify turned out to win more reward. And Anthropic's Claude 3.7 Sonnet (February 2025) folded the two modes into one system — billed by Anthropic as the first "hybrid reasoning" model, where the same model either answers instantly or, when asked, thinks visibly at length, with a budget you can set for how long. That's the ancestry of every "thinking" toggle you see today.

The practical consequence is a genuinely new dial called test-time compute. Until now, all the money in this chapter was spent before you arrived — training-time compute, frozen into the knobs. A reasoning model can also spend compute while answering you: the longer it's allowed to think — hundreds or thousands of scratch tokens, drafted, checked, revised — the better it does on hard problems. Same knobs, better answers, paid for in seconds and tokens. And it partially repairs the train-car problem: the wrong turns still happen, but on a private practice track. By the time the visible answer starts, the dead ends have been explored and abandoned where you (mostly) never see them.

Thinking toggle	Use it for	Skip it for	Why
ON (slow, thorough)	Math and anything with numbers that must be right; multi-step planning ("build the full dinner timeline around Dad's flight"); debugging; comparing options against several constraints at once	—	These tasks have wrong turns in them, and scratch paper is where wrong turns are cheap. Accuracy rises with thinking time.
OFF (fast, cheap)	—	Quick lookups, rewriting an email, brainstorming, casual chat, summarizing text you pasted	Thinking costs seconds and money, and on easy tasks it buys little — sometimes just a longer route to the same answer.

Why your chatbot doesn't know last week's news

Now several everyday quirks fall out of the pipeline for free.

The knowledge cutoff. Pretraining is a discrete event: at some point the mountain of text is frozen, and the months of knob-tuning begin. Everything the model "knows" was in that frozen snapshot. If the snapshot was frozen in March and a restaurant opened in June, the model has never seen a single token about it — asking harder won't help; the knowledge simply isn't in the knobs. That's why chatbots tell you "my knowledge has a cutoff date," and why fresh information has to be brought to the model (pasted into the chat, or fetched by the search and retrieval tools of Chapter 7) rather than found inside it.

Think of the model as an astronaut just back from a long mission: brilliant, widely read, and off-planet since launch day. Ask about anything up to launch and the answers are superb; ask about last month's election and the honest reply is "I was in orbit — tell me what happened." A pasted article or a search tool is exactly that briefing. One flag on the analogy: a real astronaut knows they've been away and reflexively says so, while a model has no felt sense of elapsed time — unless its training taught it to announce the cutoff, it may simply answer from stale knowledge with full confidence, like an astronaut who hasn't noticed the calendar.

The system prompt. One more ingredient sits between the trained model and your first message. Before the conversation starts, the provider silently prepends a system prompt — written instructions like "You are a helpful assistant. Today's date is August 11, 2026. Be concise; refuse harmful requests." It's not training — no knobs change — it's just text placed at the front of the token sequence, steering predictions the way any preceding text does (that's also how the model "knows" today's date). Training forged the instrument; the system prompt is a sticky note taped to it each morning.

The bill

Some numbers, for honest awe. Frontier-model pretraining runs on tens of thousands of GPUs for months; the total computation is measured in numbers like 10²⁵ to 10²⁶ arithmetic operations — give every human on Earth a calculator doing one sum per second, and they'd need tens to hundreds of millions of years to match. Disclosed and estimated training costs for frontier models run from tens of millions of dollars into the hundreds of millions for 2025–26 flagships (OpenAI's Sam Altman said in 2023 that GPT-4 cost more than $100 million), before counting the data centers themselves. Stages 2 and 3, by contrast, cost a small fraction of stage 1 — which is why there are only a handful of organizations that build frontier base models, but thousands that fine-tune them.

Money and chips are one way to count the bill; watts and water are another, and it's the question readers ask most: does every prompt boil a lake? In 2025, Google published what it believes is the most complete per-prompt accounting yet, for its own Gemini app: the median text prompt used 0.24 watt-hours of energy (about nine seconds of TV), 0.26 milliliters of water (about five drops), and emitted 0.03 grams of CO₂ — with efficiency improving fast (energy per median prompt fell 33× in the single year to May 2025). So no: your question does not boil a lake. Two caveats keep the number honest. First, it's self-reported and not independently verified — and Google itself points out that narrower "just the chip" accounting would show less than half the true figure, because it ignores idle machines, host computers, and cooling; whenever you see dueling AI-energy statistics, the first question is what did they count? Second, tiny-per-prompt is not tiny-in-total: billions of daily prompts, plus the far hungrier work of training, are driving a data-center building boom whose electricity and water demands are a genuinely big story. Per-prompt guilt is misplaced; grid-level attention is not.

Drawn as compute — chips × time, the thing the money mostly buys — the three stages are almost comically lopsided:

1 · Pretraining~95%+

2 · Fine-tuning~1–2%

3 · Preference~2–4%

Where to go deeper

1. Easy: "Large language models, explained with a minimum of math and jargon" by Timothy B. Lee & Sean Trott — its second half covers training: why prediction forces learning, and how human feedback reshapes behavior.
2. Medium: Andrej Karpathy, "Intro to Large Language Models" — a one-hour talk by a co-founder of OpenAI walking through exactly this pipeline (base model → fine-tuning → RLHF), with real numbers and demos. The single best hour of video on this chapter's material.
3. Deep: "Illustrating RLHF" (Hugging Face) — a diagram-first technical walkthrough of preference training: reward models, the training loop, and the failure modes. Some jargon, but you now have every concept it assumes.