What is an NPU, and how is it different from a GPU?

Think of a busy food stall. The owner can cook any dish on the menu, but she works through orders one at a time. Behind her stands a row of fifty helpers who can all chop at once, whatever the dish. In the corner sits a small machine that does nothing except slice onions, all day, on very little electricity.

Those three are the processors inside your phone. The owner is the CPU. The fifty helpers are the GPU. The onion slicer is the NPU.

An NPU is narrow on purpose. Narrow is the whole point. It gives up the ability to do anything else, and in exchange it does one job cheaply.

What an NPU is

NPU stands for neural processing unit. It is a block of circuits built to run neural networks. Those are the models behind speech recognition, photo clean-up, translation and chat.

On a phone or a laptop the NPU is rarely a separate chip. It is one section of the main chip, sitting next to the CPU and the GPU and sharing the same memory. Makers use different names for it. Neural engine and AI engine mean the same kind of block.

Silicon can be shaped this way because a semiconductor lets a designer lay down any pattern of switches. If you know the sum you will run a trillion times, you can build hardware that does only that sum.

What a neural network asks a chip to do

Strip away the mystery and a neural network is mostly arithmetic. Multiply two numbers. Add the result to a running total. Do that a few billion times. Then push the totals through a simple rule that flattens the negative ones to zero.

Nothing there is clever on its own. The work is in the volume, and in the fetching. Every number has to be pulled out of memory and put back again. Moving a number costs more energy than multiplying it. That one fact shapes the whole design.

How an NPU works, step by step

1. A model is trained somewhere else, on large machines in a data centre.
2. Software converts the trained model into a form the NPU accepts, and shrinks the numbers down to whole numbers of eight bits or fewer. That step is called quantisation.
3. The model is loaded onto the phone or laptop once, and stays there.
4. When you ask for something, the NPU pulls in a block of numbers and streams it through a grid of small multiply-and-add units.
5. Results pass sideways from one unit to the next inside the grid, so most numbers never go back to memory in between.
6. The answer returns to the app. Nothing has to leave the device.

Step five is the trick. The grid is laid out so that data flows through it. Fewer trips to memory means less energy. Less energy means the phone stays cool and the battery lasts.

NPU, GPU and CPU side by side

Part	Built for	Typical power	Usual job
CPU	Anything, a few tasks at a time	A few watts in a phone, more in a desktop	Runs the system and most apps
GPU	Very large amounts of parallel maths	Tens of watts in a laptop, hundreds in a server	Graphics, training, serving big models
NPU	Neural network maths at the lowest power	A few watts at most	Camera, voice and text features on the device

Read that as a rough guide. Real parts overlap. Modern GPUs now contain blocks built for the same neural network sums, and some laptop chips blur the line further.

Why rough numbers save power

Here is the part most explanations skip. A trained model can be stored with very exact numbers or with rough ones. The exact kind is called floating point, and a common size is 32 bits for each number. The rough kind is plain whole numbers, often 8 bits, sometimes 4.

Why does that matter? Because a smaller number is cheaper in every way. It takes less room in memory. It takes less energy to move. And the circuit that multiplies it is far smaller, so the chip can hold thousands of them side by side.

Does the model still give good answers? Usually yes, as long as the shrinking is done with care. Trained networks cope with rough numbers surprisingly well. THE POWER SAVING COMES FROM ACCEPTING ROUGH NUMBERS. That is the deal an NPU makes, and it is why it can serve a model that would leave a GPU drawing far more power.

Where you already meet one

You have used an NPU today if you carry a recent phone.

It checks your face at the lock screen. It cleans up a photo in the moment after you press the shutter, and finds the faces and the text in it. It strips background noise out of your calls. It runs voice typing and live captions. It drives the small translation and writing tools now built into keyboards.

Cars use them to read lanes and road signs. Security cameras use them to tell a person from a passing cat. A smart speaker uses a tiny one to listen for its wake word without draining the socket.

The thread running through all of that is location. The work stays on the device. Therefore it keeps working with no network, it costs no data, and the raw picture or sound never travels.

What an NPU is bad at

An NPU is quick at the job it was built for and useless at everything else.

It cannot train a model of any real size. Training wants exact numbers, huge memory and days of full power, so it stays on GPUs in data centres. An NPU also struggles with any operation its designer left out. When it meets one, the work falls back to the CPU and slows to a crawl. And it is tied to the maker’s own software tools, so a model usually has to be converted before it will run at all.

Then there is the number on the box. Makers quote TOPS, meaning trillions of operations per second. Treat that as a ceiling rather than a promise. Real speed usually depends on how fast the chip can feed itself from memory, and on how good the software is. Two chips with the same TOPS can behave very differently.

One more limit sits outside the chip. NPUs come from the same small set of advanced plants as every other leading processor, which is part of why chip fabs matter so much. Shrinking transistors no longer buys the free speed it once did, so building narrow blocks like this is now a main route to faster chips. That is the practical answer to what came after Moore’s Law.

What to check on your own phone

Do this now. Put the phone into flight mode so nothing can reach the internet. Then try voice typing a message, or ask the gallery app to find every photo with a dog in it.

If it still works, the model ran on the device. On most modern phones the NPU did that work, quietly, on a sip of battery. If it stops working, your phone is sending the job to a server, and you are paying for the data.

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.