Chapter 12
When Models Stop to Think
Every major chat app now has a "thinking" switch. This chapter is the owner's manual: what the switch actually does, when it earns its wait, and why the "thoughts" on your screen aren't quite what they seem.
Part IV closed the book's main story: you can now explain AI → ML → LLM → Agent, and you know where the machinery breaks. Part V asks a different question — not how does it work? but how do I live with it? Chatbots that think, voices that lie, images that never happened, "friends" made of tokens, agents holding your wallet, and a workplace and classroom quietly rearranging themselves. We start with the most visible change in the chat apps themselves: a little toggle, usually labeled something like Think, Reasoning, or Extended thinking. From the outside it's just a switch on the same black box. Flip it, and the answer arrives slower — and, on the right kind of question, noticeably better. What did the switch actually change?
One box, two speeds
Back to the birthday dinner one more time — this time the logistics endgame. You ask your chatbot:
"The dinner is Saturday at 7pm. The bakery needs 48 hours' notice for a custom cake and is closed Sundays and Mondays. My sister lands Friday at 6pm and wants to sign the card before it goes in the gift bag, which I'm assembling Thursday night. When do I need to do what?"
With the toggle off, a fast answer comes back in two seconds. It's fluent, confident — and if you trace it carefully, there's a decent chance something is off: the card gets signed after the bag is sealed, or the cake gets ordered Friday for Saturday, inside the 48-hour cutoff. With the toggle on, the app shows a spinner — "thinking…" — for fifteen seconds or so, sometimes with a collapsible pane of scratch-work. Then the answer walks backward from Saturday 7pm, checks the 48-hour rule against the bakery's closed days, catches the card-before-bag conflict, and proposes moving the gift-bag assembly to Friday night. (This example is illustrative, not a transcript — your model and your mileage will differ. The shape of the difference is the point.)
Chapter 6 gave you the home remedy this feature grew out of: "ask the model to reason step by step first," because tokens are the model's only scratch paper — whatever it writes down becomes context its later tokens can attend to. Chapter 5 told you what the labs did next: they stopped treating that as a prompting trick and trained it in, rewarding models for reaching checkably correct answers via free-form scratch-work. This chapter picks up where those left off: what actually happens when you flip the switch, and how to use it well.
What the toggle does, in five steps
- Your question goes into the context window — same as always. Nothing about the box's machinery changed; it's the same token loop from Chapter 4.
- The model writes scratch tokens first. Instead of starting its reply, it starts a private draft: restating the problem, listing constraints, trying an ordering, checking it. Hundreds or thousands of tokens of work you mostly don't see.
- It backtracks. This is the trained-in part. Somewhere in the scratch-work you'd see the machine equivalent of "wait — 48 hours before Saturday 7pm is Thursday 7pm, not Friday; let me redo this." Wrong turns still happen, but they happen on a practice track, not in your answer.
- The final answer is written conditioned on the scratch-work. All those draft tokens are sitting in the context window, so the visible reply can attend to the checked timeline instead of improvising one token by token.
- The app shows you the answer — plus, usually, a tidied summary of the thinking. Hold that word summary; it matters in a minute.
The dial this creates is called test-time compute, and it's genuinely new. Chapter 5's economics all happened before you showed up: millions of dollars of training, frozen into the knobs. A reasoning model can also spend compute while answering you — and on hard problems, more thinking time buys better answers, paid in seconds and (for developers) tokens. Anthropic's Claude 3.7 Sonnet, announced in February 2025 as "the first hybrid reasoning model on the market," made the trade explicit: one model that either answers instantly or, when asked, "self-reflects before answering, which improves its performance on math, physics, instruction-following, coding, and many other tasks" — with a dial, in the version programmers build on, for exactly how many thinking tokens to allow. That's the machinery behind every thinking toggle you meet today, whatever the brand.
Where the switch came from (a one-paragraph recap)
Chapter 5 told the origin story, so here it is in one breath: OpenAI's o1 (late 2024) was the first mainstream model of this kind — in its maker's words, "trained with large-scale reinforcement learning to reason using chain of thought." DeepSeek's R1 work (through its R1-Zero variant) showed the same abilities could be incentivized with pure reinforcement learning — no human-written reasoning examples — with double-checking behaviors emerging on their own, because pausing to verify won more reward. And the hybrid design folded thinking and fast modes into a single model. Within about a year, "reasoning mode" went from research headline to a default switch in every major chat product.
When to flip it — and when not to
The rule falls straight out of the mechanism. Scratch paper helps exactly when the task contains wrong turns — steps that can be checked, orderings that can clash, cases that can be missed. It buys little when the task is recall, style, or conversation.
| Flip it ON | Leave it OFF | |
|---|---|---|
| Everyday examples | The cake-timeline question; splitting a vacation bill with three currencies and one person who skipped dinner; comparing two phone plans' real annual cost; debugging why a spreadsheet formula misbehaves; "what's wrong with this argument?" | "Rewrite this email warmly"; "what's a good side dish for pad thai?"; brainstorming gift ideas; summarizing an article you pasted; casual back-and-forth. |
| Why | Multi-step, checkable, easy to fumble token-by-token. Accuracy rises with thinking time. | No wrong turns to catch. You pay seconds (and, if you're paying per token, money) for a longer route to the same answer. |
| What it costs | Seconds to minutes of waiting; more tokens. | Nothing — fast mode is the default for a reason. |
One more habit worth keeping even with the toggle on: Chapter 6's "ask for reasoning steps" advice still helps, because it shapes the visible answer — "first list the constraints, then the timeline, then flag anything tight" gets you an answer you can check, which matters more than ever. The toggle improves the odds the answer is right; structure improves your odds of noticing when it isn't.
The thoughts on screen are not the thoughts
Now the honest part, in two layers.
Layer one: what you read is usually a summary. Most apps don't show the raw scratch-work; they show a cleaned-up digest of it. Useful for spotting whether the model considered the bakery's closed days — but it's a press release about the thinking, not the thinking.
Layer two is stranger: even the raw scratch-work doesn't reliably reflect what drove the answer. In 2025, Anthropic researchers tested this directly. They slipped hints into evaluation questions — sometimes innocuous ("a Stanford professor thinks the answer is A"), sometimes compromising (the answer obtained via unauthorized access) — and checked whether models that visibly used the hint ever mentioned it in their written reasoning. Mostly, they didn't: Claude 3.7 Sonnet mentioned the hint it had used only about 25% of the time, DeepSeek R1 about 39%. Worse, in experiments where models learned to exploit a scoring loophole, they exploited it almost constantly while almost never saying so — often writing a plausible-sounding justification for the answer instead. The researchers' own summary: advanced reasoning models "very often hide their true thought processes."
Nothing spooky is going on — no secret inner monologue is being concealed on purpose. Remember what the scratch pad is: more predicted tokens. The model produces reasoning-shaped text because reasoning-shaped text was rewarded, and text that looks like a clean justification can score well even when the actual computation took a shortcut. The practical rule: read the thinking pane to check coverage — did it consider the right things? — but audit the answer, not the essay. A tidy rationale is not evidence of a tidy process; you verify the timeline against the bakery's hours, same as ever.
Misconception: "Thinking mode fixes hallucination"
No — and on some measurements the arrow briefly pointed the wrong way: Chapter 6's Simplified! box has OpenAI's own numbers showing a 2025 reasoning model hallucinating more on a people-facts test than its predecessor, not less. Reasoning training rewards checkably correct answers on problems that have checkers — math, code, logistics. "What year did that restaurant open?" has no checker inside the model; it's still next-token recall from thin data, thought about at length. Thinking mode makes the model better at problems with an answer key. It does not install a truth detector, so the verification habits from Chapters 6 and 10 stay in force with the toggle on.
Simplified!
We drew the scratch pad as "private," but the privacy is a product choice, not a property of the machine: the scratch tokens sit in the same context window as everything else, and some products show them nearly raw while others summarize or hide them. We also said "the model checks its work" — really, it produces tokens that function as checking because checking-shaped detours won reward during training. There is no separate verifier module inside; it's the one token loop, steered into a useful ritual. That's why the ritual sometimes goes through the motions — long, diligent-looking scratch-work arriving at a wrong answer with full ceremony.
Try it now (5 minutes)
Take the cake-timeline question from this chapter — or better, a real scheduling tangle from your own week ("the car's at the shop until Thursday, the recital is Friday at 5, groceries need to happen…"). Ask it twice in the same app: once with thinking off, once with thinking on. Time both. Then check both answers yourself against the constraints. Three things to notice: whether the fast answer contains a subtle ordering mistake, whether the thinking pane mentions the constraint that matters most, and whether the slow answer was worth the wait for this particular question. That last judgment — is this task hard enough to pay seconds for? — is the skill this chapter is trying to give you.
Where to go deeper
- Easy: Anthropic's Claude 3.7 Sonnet announcement — the first hybrid reasoning model, explained by its makers in plain product terms: visible extended thinking, the thinking budget, and the trade of speed for quality.
- Medium: "Reasoning models don't always say what they think" (Anthropic research) — the hint experiments behind this chapter's faithfulness numbers, written accessibly; it will permanently change how you read a thinking pane.
- Deep: The OpenAI o1 System Card — the safety report for the first mainstream reasoning model, including how large-scale reinforcement learning on chain of thought changes both capabilities and risks.