A stamp full of switches

Every AI answer is worked out on a chip in a datacenter. The best known is NVIDIA’s H100. Its working heart, called the die, is about 28.5 millimeters on a side, barely bigger than a postage stamp. On it sit 80 billion transistors, tiny switches that do the arithmetic.

That stamp is close to the biggest rectangle a chipmaking machine can print in one go. To go further, NVIDIA’s 2024 B200 joins two dies of about that size with a fast link so they behave as one chip. Together they hold 208 billion transistors.

The plate around the chip

The die never works alone. Think of it as the main course in the middle of a dinner plate, with the side dishes arranged around it. The side dishes are stacks of fast memory called HBM, where the model’s weights are kept.

A stack can be a pile of twelve memory chips, and the whole pile is still thinner than a credit card. The stacks sit a few millimeters from the die and talk to it over 1,024 data wires each. A memory stick in an ordinary computer has 64.

Why memory sets the typing speed

To write each new token, the chip must read every one of the model’s weights out of memory once.

A small model with 8 billion weights takes about 16 gigabytes of memory. The usual server version of the H100 can read 3,350 gigabytes a second. Divide one by the other and you get about 209 full reads a second, so about 209 tokens a second for one conversation. Real systems reach 60 to 80 percent of that. For one conversation, memory sets the speed limit.

Building your own

Some of the biggest buyers of AI chips also design their own. On October 12, 2026, Microsoft announced Maia 300, the next chip in its Maia family, due in production in 2027.

Microsoft says each chip will carry eight stacks of the newer HBM4 memory, with 18 terabytes a second of peak bandwidth, within 950 watts. It also claims 40% more tokens per megawatt than leading GPU-based systems. That measures how much text a system writes for a given amount of electricity.

These are Microsoft’s own projections for a chip that is not yet in production. Yahoo Finance described the chip as part of an effort to rely less on Nvidia and AMD.

In one sentence: an AI chip is a stamp-sized grid of switches ringed by fast memory, and how quickly it can read that memory sets how fast and how cheaply it writes.