Chapter 6
Talking to LLMs Well
You now know what's inside the box. This chapter turns that knowledge into skill: memory, randomness, hallucination, and prompts that actually work.
From the outside, the black box seems moody. Some days it nails the dinner plan on the first try; other days it forgets what you told it, invents a restaurant that closed in 2019, or gives your sister a completely different answer to "the same" question. None of that is mood. Every one of those behaviors falls straight out of the machinery you've just learned — the token loop, the probability table, the frozen knobs. This chapter walks through four mechanisms (memory, wording sensitivity, randomness, hallucination) and ends with the prompting patterns they imply. By the end, "talking to LLMs well" won't be a bag of tricks. It'll be obvious.
The context window: all the memory there is
Chapter 5 left the model's knobs frozen at the factory. So how does it remember that you said Mom loves Thai food? Simple — maybe disappointingly simple: the entire conversation so far is re-sent through the model every single turn. Your messages, its replies, the system prompt — all of it is the "text so far" that the token loop conditions on. The model has no other memory. What looks like remembering is just re-reading.
The amount of text the model can take in at once is called the context window — its short-term memory, measured in tokens. Toy number first: imagine a model that could only see the last 10 tokens. Say "Mom loves Thai" and then chat for three more sentences, and the fact literally scrolls out of view — unseeable, gone. Frontier models' 2026 windows run from about 200,000 tokens to a million or more (a long novel, or several), though chat apps may give you less, especially on free plans, so scrolling out is rarer than it used to be — but the principle is unchanged: if it's not in the window, it does not exist for the model. And long before the hard limit, quality degrades: in very long chats, models get demonstrably worse at using details buried in the middle. A fact on page 2 of a 300-page conversation is technically visible and practically fuzzy.
The best mental image is a whiteboard of fixed size. Everything the model can possibly consider is what's currently written on the board — your messages, its replies, the system prompt in the corner. When the board fills up, the only way to write new notes is to erase the oldest ones, and the model doesn't feel the eraser: it simply has no idea those notes ever existed. One flag on the analogy: a person who filled a whiteboard would remember having erased something and might ask you to repeat it; the model can't, because for it the erased text was never there at all.
Three practical consequences:
- New chats start from zero. A fresh conversation is an empty window (plus the system prompt). Yesterday's brilliant dinner discussion is gone unless your app has a bolt-on memory feature (Chapter 5) or you re-paste it.
- The "start a new chat" tactic. When a conversation goes off the rails — the model keeps apologizing, or clings to a wrong assumption — don't argue with it. Remember the train cars: every bad turn is still in the window, feeding the next prediction. The most probable continuation of a conversation full of confusion is… more confusion. Open a fresh chat, restate the task cleanly with only the good context, and you've replaced a polluted window with a clean one. This one tactic fixes more stuck sessions than any clever phrasing.
- You are the memory manager. Anything you want reliably used — Mom's shellfish allergy, the $200 budget, the date — belongs in the window, stated explicitly, ideally recently.
The butterfly effect of wording
You and your sister ask "the same" question:
- You: "Suggest a restaurant for my mom's birthday dinner."
- Her: "Where should I take my mother for her birthday meal?"
Different answers come back, and it feels arbitrary. But recall the loop: every token is predicted given all the text so far. Different words in → different vectors → different attention patterns → a different probability table for token one. And whichever token gets picked becomes input for token two — so a small early divergence compounds, token by token, into visibly different answers. "Mom" and "mother" even carry different statistical auras in training text (casual vs. formal), nudging tone before content. Tiny cause, cascading effect: a butterfly flapping in the prompt.
Two lessons. First, don't over-read any single answer — you sampled one path through a probability tree, not the answer. Second, this is a feature: if an answer disappoints, rephrasing genuinely re-rolls the dice with different odds, and asking twice and comparing is a legitimate quality check for anything factual.
Temperature: the randomness dial
Even the exact same prompt in two fresh chats usually yields different answers — because step 5 of the loop ("pick one") deliberately doesn't always take the top token. A setting called temperature controls how adventurous the pick is: near 0, always take the most probable token; higher, give the underdogs a real chance. Here's the same prompt at three settings (precomputed illustrations, not live output — chat apps don't publish their setting or expose the dial; developer tools do, and OpenAI's and Anthropic's APIs default to 1):
| Temperature | Prompt: "Suggest a birthday dinner for Mom" | Character |
|---|---|---|
| 0 (always the top token) | "A classic choice is dinner at an Italian restaurant. Book a table, order a cake for dessert, and consider bringing flowers. If she prefers a home-cooked meal, make her favorite dish." | Safe, generic, repeatable — run it again, get nearly the same words. Good for facts, extraction, anything with one right answer. |
| 0.7 (a middle setting) | "How about a Thai place with a quiet back room? Book for 7pm, ask them to bring out mango sticky rice with a candle, and have everyone write one favorite memory of her on the back of the menu." | Coherent but varied — a different plausible plan each run. The kind of balance chat apps aim for. |
| 1.5 (high) | "Rent the little ferry that crosses the harbor at dusk — dinner unfolds in courses at each stop: soup dumplings portside, her 1974 honeymoon playlist, and a cake shaped like the crossword she does every…" | Surprising, sometimes wonderful, sometimes unraveling mid-sentence. Underdog tokens win often; each win steers the train somewhere stranger. |
Hallucination: why the box makes things up
Ask about Thai restaurants in your small hometown and you may get "Golden Orchid Thai on Main Street — locals love the crispy duck." Persuasive. Possibly nonexistent. This is hallucination — the model stating falsehoods with full confidence — and it isn't lying or malfunctioning. It's the token loop doing exactly its job.
Walk the mechanism. The model must always emit a plausible next token — that is the only operation it has. There is no step where it checks a database (there isn't one, remember) and no built-in "actually, I don't know" detector. When the topic is thick in training data — the capital of France — the plausible token and the true token are the same, because the truth appears thousands of times. When the topic is thin — restaurants in a small town, an obscure paper, a minor legal citation — there aren't enough training examples to make the true answer the probable one. But text shaped like an answer is still highly probable: restaurant reviews have a rhythm, and "Golden Orchid" fits it beautifully. Plausible ≠ true, and when data is thin, plausible-but-false wins. The fluency never wavers, because fluency comes from the abundant patterns of how such text sounds, not from the scarce facts inside it — which is exactly why hallucinations are so convincing.
Risk, then, tracks how thin the topic is and whether a wrong answer even exists:
| Risk | Task type | Why |
|---|---|---|
| High | Citations and quotes; specific numbers, dates, prices; niche facts (small-town restaurants, obscure people); anything after the knowledge cutoff | Thin training data + a strong template for what answers look like = fluent invention. Verify anything load-bearing. |
| Low | Rewriting and editing your text; brainstorming; summarizing text you pasted in; explaining mainstream concepts; drafting from details you supplied | Either the facts are sitting in the context window (attention can fetch them) or there's no single fact to get wrong. |
Note the deepest pattern in that table: the model is far more reliable working on text you gave it than recalling text from training. Supplying the menu beats asking it to remember the menu — an idea so useful it gets its own chapter (RAG, Chapter 7).
Simplified!
"Thin training data" isn't the whole story. Models also hallucinate because training rewarded it: a base model is graded on plausible continuation, and both raters and benchmarks historically favored confident answers over "I don't know" — so guessing was, statistically, the winning move. This isn't just this book's intuition anymore: a 2025 OpenAI research paper ("Why Language Models Hallucinate") formalized exactly this, arguing models "are optimized to be good test-takers, and guessing when uncertain improves test performance," and calling the way standard benchmarks penalize honest uncertainty an "epidemic" — the proposed fix being to change how the tests are scored, not just the models. Labs now train models to decline and hedge more, and scores on many benchmarks have improved. But don't assume newer means safer: OpenAI's own measurements found its 2025 reasoning model o3 hallucinated on a people-facts test at twice the rate of its predecessor o1 (33% vs. 16%, with the smaller o4-mini at 48%) — it made more claims overall, right ones and invented ones, and "more research is needed to understand the cause." The trend is jagged, not steadily improving, which is why the verification habit stays regardless of what model you're on.
Prompting patterns that fall out of the machinery
Everything above compresses into six habits. None is magic; each just puts better text in the window for the probability machine to condition on.
- Give context. The model can't attend to what isn't in the window. Weak: "Suggest a restaurant for Mom's birthday." Strong: "Suggest a restaurant for my mom's 60th birthday in Portland: she loves Thai, is allergic to shellfish, hates loud rooms; 6 people, under $200, next Saturday." Every clause redirects attention and prunes whole branches of the probability tree.
- Show an example of the output you want. The loop continues patterns, so hand it one: "List options in this format — Name | cuisine | noise level | est. cost | one reason Mom would love it." One example row beats three sentences of description.
- Ask for reasoning steps. "Before recommending, first list what matters for this dinner, then evaluate three candidate styles against those criteria, then pick." Tokens are the model's only scratch paper: reasoning it writes out becomes context its later tokens attend to. Criteria written first make the final pick conditioned on them. (Modern "reasoning" models were trained to do a version of this internally — that's Chapter 5's fourth stage, and the "thinking" toggle is how you request it. Asking still helps structure the visible answer.)
- Assign a role. "You're a restaurant concierge who has planned hundreds of milestone birthdays." Roles aren't costume parties — they steer the model into the region of its training distribution where the good answers live: concierge-flavored text is denser in booking tactics and menu wisdom than generic text. (Roles shape tone and focus; studies find they do little for factual accuracy.)
- Iterate — never accept draft one. Draft one is one sample from the tree. Reply with steering: "Closer to option 2, but cheaper, and she'd hate a tasting menu — swap that." Each correction lands in the window and sharpens every subsequent prediction. Three quick rounds beat one perfect mega-prompt, and iterating is how the pros actually work.
- Push out of the probable zone for originality. Default output is, by construction, the most statistically typical answer — the Italian-restaurant-and-flowers zone. For originality, forbid the typical and force the underdogs: "Give 5 ideas, none involving a restaurant reservation. Make #5 something a party planner would call risky." You're doing manually what high temperature does randomly — visiting unusual probability space on purpose, with you as the filter.
Here's what the habits look like applied — three makeovers of prompts from the dinner project. In each row, the strong version isn't longer for its own sake; every added clause is doing a specific mechanical job in the window:
| Weak prompt | Strong prompt | Why the strong one works |
|---|---|---|
| "Suggest a restaurant." | "Suggest 3 cozy Thai places in Capitol Hill under $50/person. My mom dislikes loud rooms — explain each pick." | Context prunes the probability tree: Thai, Capitol Hill, $50 each eliminate millions of generic continuations, 3 and explain each pick set the output pattern, and dislikes loud rooms gives attention a criterion to check every candidate against. |
| "Write a birthday toast for my mom." | "Write a 60-second toast for my mom's 60th. Warm but funny, no clichés about aging. Work in: her pad thai obsession, the Lisbon trip, teaching me to drive. End on the line 'here's to the next adventure.'" | Supplied details beat recalled ones — the model can't hallucinate memories you handed it. Length, tone, and a fixed closing line constrain the shape, and no clichés about aging explicitly fences off the most probable (and most groan-worthy) region of the tree. |
| "Is this restaurant good?" | "Here's the menu and 6 recent reviews [pasted]. Based only on this text: rate fit for a quiet 6-person birthday, flag anything with shellfish, and note prices over $25. If the reviews don't say, say 'not mentioned.'" | Everything load-bearing sits in the window, where attention can fetch it, instead of in thin training data, where plausible invention lives. Based only on this and say "not mentioned" give the model a high-probability honest exit — without one, something confident is the likelier continuation. |
Misconception: "Prompting is magic incantations"
You'll see threads hawking "the 7 secret prompts they don't want you to know," with mystical phrasings to copy verbatim. Now you know better: there is no incantation parser inside — only a probability machine conditioning on whatever text is in the window. Prompts work exactly to the extent that they add context, set patterns, and steer attention; the six habits above are the whole "secret." Corollary: prompt quality has a ceiling. No phrasing adds knowledge that isn't in the knobs or the window — a perfect prompt about post-cutoff news still yields a hallucination. Steering beats spells; supplying beats steering.
Try it now (5 minutes)
A three-part experiment. (1) Butterfly: ask "Suggest a restaurant for my mom's birthday dinner" in two separate fresh chats — same words — and compare answers; that's temperature. Then change "mom" to "mother" in a third chat and see what shifts; that's wording. (2) Context: ask a fourth time with the full strong prompt from pattern 1 above and watch generic turn specific. (3) Hallucination hunt: ask for three Thai restaurants in a small town you know well, with signature dishes — then check. Anything invented? You just calibrated your own trust, on home turf.
Key idea
Four mechanisms explain nearly everything about talking to LLMs: the context window is the only memory (manage it; restart when polluted), wording shifts every downstream probability (rephrase and re-roll), temperature sets how adventurously tokens are picked, and hallucination is plausible-beats-true on thin data (verify the high-risk tasks, relax on the low-risk ones). Good prompting is just engineering the window: context, examples, reasoning room, a role, iteration, and deliberate trips off the beaten probability path.
Where to go deeper
- Easy: "Thinking Like an AI" by Ethan Mollick — a short essay on building working intuition for token prediction, context windows, and why the same prompt lands differently; the source of several habits in this chapter.
- Medium: Transformer Explainer (Georgia Tech's Polo Club) — this time, use the temperature slider: type a prompt, drag from 0 toward 2, and watch the step-4 probability bars flatten and the picks get stranger. This chapter's temperature figure, live.
- Deep: Anthropic's prompt engineering guide — the professional version of our six patterns, with worked examples (roles, examples, chain-of-thought, iteration) written for people building on LLMs, yet readable with just this book behind you.