A phone’s AI speed is capped near 19 tokens a second by memory

A review of AI chips shows memory bandwidth, not arithmetic, sets the speed of language models. A flagship phone tops out near 19 tokens a second, and an H100 near 24 on a 70 billion parameter model.

What the paper found

A review of AI hardware finds that memory, not arithmetic, limits large language models. Dividing memory bandwidth by model size gives a hard speed ceiling: about 19 tokens a second for a 4-bit 8 billion parameter model on a flagship phone, and 24 tokens a second for a 16-bit 70 billion parameter model on an NVIDIA H100.

By Mark Alex · September 9, 2026

Grade 5 reading level5 min readPreprint · not yet peer reviewed

What happened

Picture a very fast cook. She can chop faster than anyone alive. But the fridge is down a long corridor. Every onion means a walk. Soon the walking, not the chopping, sets the pace of dinner.

That is what has happened to the chips that run AI. They can multiply numbers at a stunning rate. But the model has to be fetched from memory first, over and over, one word at a time. The fetching is now the slow part.

A new review says this plainly. For large language models, the memory system has become the computer. It gathers the specifications of the main chips and works out what that limit really costs you.

The test

There is no experiment. This is a review paper called “AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall”. Siddharth Patel and Rohit Singh wrote it. Both are in the Department of Electronics Engineering at Shiv Nadar Institution of Eminence in Delhi-NCR, India. It went on arXiv on 28 August 2026 as a preprint. No journal or conference is named.

The two authors survey every main kind of AI chip. Ordinary graphics cards. Custom chips such as TPUs, Trainium, Groq and Cerebras. Reconfigurable chips. Designs that put the maths inside the memory itself. Brain-inspired and light-based chips.

Then they apply one simple sum to all of them. Take the memory bandwidth of the chip. Divide it by the size of the model. That gives the fastest possible speed for writing one answer, one word at a time. It is an upper bound, and no real system beats it.

The result

Start with the phone in your pocket. A flagship phone moves about 0.077 terabytes per second. Running a small 8 billion parameter model squeezed to 4 bits, its ceiling is about 19 tokens a second. A token is roughly a short word. The AI chip inside the phone is not the limit. The memory is.

Now the big iron. An NVIDIA H100 moves 3.35 terabytes per second. Running a 70 billion parameter model at 16 bits, its ceiling is 24 tokens a second for one user. Squeeze that same model to 4 bits and the ceiling jumps to 96. Nothing about the chip changed. There was simply less to read.

The gap shows in energy too. Reading 64 bits from main memory costs roughly 1,300 to 2,600 picojoules. Doing a sum on the chip costs a few. That is about a thousand times more to move the number than to use it.

The trend is going the wrong way. From the A100 to the B200, raw arithmetic power grew about seven times. Memory bandwidth grew about four times.

What it means

The practical rule is short. MAKE THE MODEL SMALLER BEFORE YOU BUY A FASTER CHIP. Cutting a model from 16 bits to 4 bits quadrupled the ceiling in the example above. No new hardware was needed.

It also means the number on the box lies. Chip makers sell trillions of operations per second. The review says those figures do not predict how fast a model will answer, because the chip spends its time waiting for memory.

There is a third thing worth knowing. The memory of a chat, called the key-value cache, grows as the conversation gets longer. For a 175 billion parameter model at a very long context, the review puts that cache at 618 gigabytes. The model weights are 350 gigabytes. The conversation ends up bigger than the model.

Business ideas from this paper

A will-it-run checker. Pick your phone and pick a model, and the tool gives the realistic speed from memory bandwidth rather than from the chip maker’s headline number. Who buys it: app developers deciding whether to put AI on the device or in the cloud. A price to test: 15 dollars a month per developer seat. A one-week test: publish free results for 20 popular phones, ask for an email to compare more than three, and count the emails.
A model shrinking service. You take a small firm’s model, cut it to 4 bits, measure the speed before and after on their actual device, and hand back both numbers. Who buys it: small software firms whose on-device AI feature is too slow to ship. A price to test: 2,500 dollars per model. A one-week test: offer it to five firms with a slow AI feature and count how many send you a model.
Bandwidth-first buying advice. Most buyers compare AI servers on operations per second. You rank them on bandwidth per dollar for the exact model they plan to run. Who buys it: technology buyers at mid-sized firms about to spend on AI servers. A price to test: 1,000 dollars per purchase review. A one-week test: publish a short table of bandwidth per dollar for the common chips and count who writes in asking about their own shortlist.

How sure can you be?

Read the ceilings as ceilings. The authors are clear that these are idealised upper bounds. They assume one user at a time, they count only the model weights, and they ignore the growing chat cache, the attention maths, scheduling and communication between chips. Real speeds come in lower.

Most of the input is vendor specification, not measurement by the authors. They say so, and they warn that the results they collected are not comparable with each other, because each one uses a different model, batch size and operation.

THIS IS A REVIEW, NOT AN EXPERIMENT. Two authors at one institution. No journal or conference is named. No code or dataset comes with it. The choice of what to include is theirs.

What would settle the picture is a standard, published benchmark run of the same model on each of these chips, measured end to end, with energy per token reported next to speed. The review points at exactly that gap.

Do this today

Before you buy anything to run AI, do one division. Take the memory bandwidth of the device and divide it by the size of the model you plan to run. That answer is your speed ceiling, and no software will beat it.

Source: AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall, August 2026. arXiv:2608.28048 · arxiv.org (preprint · not yet peer reviewed).

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.

Questions people ask

what is the memory wall?

It is the gap between how fast a chip can compute and how fast it can be fed with data. Modern AI chips can multiply far faster than memory can deliver the numbers. The review argues this gap, not arithmetic, is now the main limit on running large language models.

why does shrinking a model make it faster?

Because the chip has to read the whole model from memory for every token it writes. A smaller model means less to read. In the review's table, cutting a 70 billion parameter model from 16 bits to 4 bits raises the H100 ceiling from 24 tokens a second to 96.

does a higher TOPS number mean a faster AI chip?

Not for running language models. The review says peak operations per second and operations per watt are nameplate figures that mislead, because the chip sits waiting on memory during the phase that writes the answer. Standard benchmarks exist because the spec sheet does not predict delivered speed.

what is the key-value cache?

It is the running memory of a conversation that the model re-reads at every step. It grows with the length of the chat. The review puts it at 618 gigabytes for a 175 billion parameter model at a very long context, larger than that model's 350 gigabytes of weights.