The sealed box

Start, as always, with the black box. Everything you've learned so far describes an LLM as a text-in, text-out machine: you type words, it predicts words back. That box is astonishing — but notice three things it cannot do, no matter how clever it is.

First, it's frozen in time. An LLM learned everything it knows during training, which ended at a specific date — its knowledge cutoff (the last day of data it was trained on, which you met in Chapter 5). Ask a bare model "what's a good new Thai restaurant near me?" and it can only answer from a snapshot of the internet that might be a year old. Restaurants open, close, and change menus; the model has no idea.

Second, it can't see your stuff. Your calendar, your email, the PDF of Mom's dietary restrictions your sister sent — none of it exists inside the box unless you paste it in.

Third, it can't act. It produces text. It cannot look up a live website, check the weather, or book a table. It can only describe those things — sometimes convincingly enough that people believe it actually did them.

So let's return to our running example. You ask a bare LLM: "Where should I take Mom for her birthday dinner on Saturday? She loves Thai food." It will give you a plausible, well-written answer — naming restaurants that may have closed in 2024, quoting prices from its training data, and inventing a "beloved mango sticky rice" if the sentence needs one. Chapter 6 explained why: it's predicting likely text, not consulting a database.

This chapter is about the three upgrades that fix this — and every one of them works the same way: the box stays a text-predictor; we just get smarter about what goes in and what we do with what comes out.

Upgrade 1: RAG — the open-book exam

The first upgrade has an intimidating name: RAG, short for retrieval-augmented generation. The idea behind it is one sentence: instead of making the model answer from memory, fetch the relevant documents first and paste them into the prompt.

Think of it as the difference between a closed-book and an open-book exam. A closed-book exam tests what you memorized — and under pressure, people misremember and improvise. An open-book exam lets you look up the actual page and quote it. Same student, dramatically fewer confident errors.

Here's the whole flow, in four numbered steps:

1. You ask a question. "Which Thai restaurants near me have good vegetarian options under $25 a plate?"
2. A search runs first. Before the LLM sees anything, a separate search system looks through a document store — a collection of texts such as restaurant menus, review sites, or your company's files — and finds the passages most related to your question.
3. The top passages get pasted into the prompt. The system quietly builds a longer prompt: "Here are excerpts from three restaurant menus and twelve recent reviews. Using only this material, answer the user's question: …"
4. The LLM answers from the pasted text, ideally citing which passage each claim came from — so you can check.

For the dinner task, RAG changes everything. Instead of a confident guess assembled from stale training data, the assistant now works from actual, current menus and reviews: "Baan Siam's menu (updated last week) lists six vegetarian mains under $25, and three reviews from this month mention the quiet back room." Every claim traces to a document you could open yourself.

To make the difference concrete, here are four questions from the dinner project, answered both ways. Watch why each answer differs — the pattern is always the same: memory versus evidence.

Dinner question	Bare LLM (closed-book)	With RAG or tools (open-book)	Why the difference
"Is Thai Orchid open on Mondays?"	Guesses from training data — may confidently describe hours for a restaurant that closed in March	A tool checks the live listing: "Permanently closed as of March 2026"	Opening hours change constantly; training data is a snapshot frozen at the cutoff
"Does Baan Siam have vegetarian mains under $25?"	Plausible invention — it has seen thousands of Thai menus and blends them into a generic one	The actual menu is retrieved and pasted in: six vegetarian mains, prices quoted verbatim	The model never saw this menu in training; retrieval puts the real one in the prompt
"What's the total for four people including tip?"	Arithmetic by word-prediction — often right, quietly wrong just often enough to burn you	A calculator tool computes 4 × $40 × 1.20 = $192, exactly	Next-word prediction imitates arithmetic; a calculator performs it
"Will Saturday be warm enough for the patio?"	Cannot know — at best a seasonal average dressed up as a forecast	A weather tool returns Saturday's actual forecast: 24°C, clear	The future isn't in any training set; only a live lookup can answer

Because the search system chooses the pages, RAG's quality lives or dies with retrieval — so think of the retriever as a librarian. A great librarian makes anyone look smart: ask a vague question and they hand you exactly the right three books, and suddenly your essay writes itself. A bad librarian hands you confident nonsense-adjacent material — books on the right shelf but the wrong topic — and you write a fluent, well-cited, wrong essay. (The analogy breaks in one way worth flagging: a human librarian will tell you "we don't have anything on that"; a retrieval system almost always returns something, relevant or not, and the model will gamely answer from whatever it got.)

One honest caveat: RAG reduces hallucination; it does not eliminate it. The model is still a next-word predictor. It can misread a retrieved passage, blend two menus together, or smooth over a gap with something plausible. What RAG genuinely gives you is (a) far fewer inventions, because the true answer is usually sitting right there in the prompt, and (b) checkability — citations you can click. It moves the failure mode from "made it up" toward "misquoted something real," which is a much better class of problem.

Upgrade 2: Tool calling — the model writes orders, the harness executes them

RAG solves "the model can't see fresh documents." But the dinner plan needs more than reading. You'd like to check Saturday's weather — Mom loves patio seating. A bare LLM cannot check the weather. It can't run code, can't browse, can't touch anything. So how do modern assistants do it?

The trick is called tool calling (sometimes function calling), and it's more mundane — and more clever — than it sounds. The model is told, in its prompt, "here are some tools you may use: get_weather(day), search_web(query), check_calendar(date). If you want one, say so in this exact format." Then:

1. You ask: "Will Saturday work for patio seating at a Thai place?"
2. The model emits a structured request — not an answer, just text in an agreed format, something like: call get_weather("Saturday"). Remember: it's still only producing text. It has asked, not acted.
3. An external program — the harness — spots that request and actually runs it. The harness (the ordinary, non-AI software wrapped around the model) calls a real weather service and gets back: "Saturday: 24°C, clear, light breeze."
4. The result is pasted back into the conversation, and the model continues: "Saturday looks perfect for a patio — 24°C and clear. Baan Siam's back terrace would be ideal."

This is worth sitting with, because it reframes how you see every fancy AI product: the LLM writes orders; the harness executes them. Like a head chef who never leaves the kitchen — she shouts "table six needs the check!" and trusts the waitstaff to actually do it. When your chatbot "searches the web," "runs code," or "checks your calendar," the model itself did none of those things. It wrote a request; a boring, reliable piece of ordinary software did the deed and reported back.

Another way to hold the same idea: the model is a chef who cannot leave the kitchen, but can send runners to the market with precise shopping lists. "Two kilos of lemongrass, the freshest you can find, from the Tuesday stall" — the chef never sees the market, but if the list is precise, the right ingredients come back, and the cooking (the language work) stays entirely hers. The precision matters: a vague list ("get some vegetables") returns the wrong ingredients, just as a badly formed tool request returns useless results. Where this analogy breaks: a human runner exercises judgment and might substitute galangal when ginger is out; the harness exercises none — it executes exactly what was written, mistakes included, which is precisely why tool requests use a rigid, agreed format.

Since different products expose different toolboxes, it helps to see the usual suspects side by side — what each tool does, and the tell-tale moment when a model reaches for it:

Tool	What it does	When the model reaches for it	Dinner-story example
Web search	Fetches live pages from the internet	The question involves anything after the knowledge cutoff, or anything local and changeable	"Which Thai places near home are open this Saturday?"
Calculator / code runner	Performs exact arithmetic, or runs small programs for real	Numbers must be right, not just plausible — totals, conversions, date math	"Four mains at $40 plus 20% tip — are we under the $200 budget?"
Calendar	Reads (and with permission, writes) your schedule	The task depends on your private commitments, which no training data contains	Spots that Dad's flight lands at 6:30pm Saturday — 7pm dinner won't work
Database / document lookup	Queries a specific, private store of records — RAG as a callable tool	The answer lives in a particular known place: menus, order history, company files	"Does Baan Siam's current menu list khao soi?"
Messaging / booking (action tools)	Changes the world: sends, reserves, pays	Only when asked — and well-built systems confirm with you first	Drafts the 8pm reservation for four, then waits for your go-ahead

Notice something elegant: RAG and tool calling are cousins. RAG pastes documents into the prompt before the model runs; tool calling lets the model request what gets pasted in, mid-conversation. In fact, modern systems often implement RAG as a tool — a search_documents() tool the model can call when it decides it needs to look something up. Both upgrades obey the same law from Chapter 6: the only way to inform an LLM is to put text in its context window.

Upgrade 3: Multimodality — pictures and voices in, sometimes out

The third upgrade is the shortest to explain. Most frontier models today are multimodal — meaning they accept more than one mode of input: images, audio, sometimes video, not just text. Under the hood, an image is converted into the same kind of number-lists (Chapter 3) that words become, so the model can attend to a photo's contents the way it attends to words in a sentence. Many models can also generate images or speech, not just consume them.

Back to the dinner. Mom texts you a photo from her trip to Bangkok years ago — a dish she adored but never learned the name of. You forward the photo to your assistant: "What is this, and do any of our candidate restaurants serve it?" The model looks at the image, identifies it as khao soi (a northern Thai coconut-curry noodle soup), and — via a tool call to search the retrieved menus — reports that Baan Siam serves it on weekends. That single exchange used all three upgrades: multimodal input (the photo), retrieval (the menus), and a tool call (the search).

The multimodal upgrade most people actually meet first, though, isn't a photo — it's talking. Every major assistant now ships a real-time voice mode (ChatGPT's voice mode; Google's Gemini Live, launched back in 2024): you speak, it speaks back, you can interrupt it mid-sentence to redirect, and on a phone the conversation keeps going hands-free with the screen locked. It feels categorically different from typing — more like a phone call than a chat — which makes it worth saying plainly that nothing categorical changed. Your speech becomes audio tokens (Chapter 3's everything-becomes-number-lists, applied to sound), the same prediction loop runs, and the reply comes back as generated audio. The warmth in the voice is synthesized the same way the words are: pattern by pattern. Voice is the best interface yet for using everything in this chapter while cooking, driving, or walking — ask it to check the weather and talk through the dinner plan aloud. It's also, as Chapter 10 will show, the modality where AI deception now hits ordinary families hardest, because the same technology that gives your assistant a voice can clone your relatives'.

The upgraded box

Zoom back out to the black box one last time. It's still a text-predictor — nothing inside the orange box changed in this chapter. But it now sits in the middle of a small support system: a retriever that hands it fresh documents, a harness that executes its requests, and inputs that can include photos and sound. Our dinner assistant can now read real menus, check real weather, and recognize a real dish.

But notice who's still doing the driving: you. You asked for the weather. You forwarded the photo. You decided what to check next. Each upgrade fired once, when you invoked it. The obvious next question is: what if the model could decide for itself which tools to use, in what order, over and over, until the whole dinner is booked? That — precisely that — is what an agent is. Turn the page.

Where to go next

Three resources, easy to deep:

1. Ethan Mollick — One Useful Thing (oneusefulthing.org). His posts on "connected" AI show, with hands-on examples, how much of a chatbot's usefulness comes from search and tools rather than raw model smarts. Zero jargon.
2. Simon Willison's blog (simonwillison.net). Willison live-documents tool calling, RAG, and multimodal features as vendors ship them, with honest notes on what actually works. Excellent for calibrating hype.
3. Anthropic — "Building Effective Agents" (anthropic.com/research/building-effective-agents). The engineering essay behind the next chapter. Its opening sections on "the augmented LLM" — an LLM plus retrieval, tools, and memory — are exactly this chapter, from the builder's side of the counter.