what does the audit actually produce?

Three lists. Features that ship now on the client's real device mix, features that need the model shrunk first, and features that belong in the cloud. Each one shows the length of answer it must write and the speed ceiling on the slower half of their phones.

is battery part of this?

Yes. Reading 64 bits from main memory costs roughly 1,300 to 2,600 picojoules against a few for a sum on the chip, so moving data dominates. A feature that runs constantly is a battery problem before it is a speed problem.

how strong is the evidence?

We score it 2 out of 10. It is a review with no new experiment, by two authors at one institution, with no journal named and no code. Most inputs are vendor specifications, so treat the rule as a shape and prove the numbers yourself.

Sell app teams an audit of what AI fits on the phone

Q: why measure phones if the sum already gives an answer?

Because the sum gives a ceiling, not a speed. It assumes one user, counts only the model weights and leaves out the growing chat memory, the attention maths and scheduling. Real devices come in lower, and reporting your own measured gap is what makes the report trustworthy.

The finding this rests on

A review of AI chips makes one point. Memory, not arithmetic, sets how fast a language model writes. The chip can multiply far faster than memory can feed it, so the chip waits.

The sum is simple. Take the memory bandwidth of the device. Divide it by the size of the model. That is the fastest the device can write, one word at a time. A flagship phone moves about 0.077 terabytes per second. Running an 8 billion parameter model squeezed to 4 bits, its ceiling is about 19 tokens a second. A token is roughly a short word. The full article is at a phone’s AI speed is capped near 19 tokens a second by memory.

Two more numbers matter for this business. Shrinking a model helps more than a faster chip, because there is less to read. An H100 running a 70 billion parameter model at 16 bits has a ceiling of 24 tokens a second, and 96 at 4 bits. And moving data is expensive. Reading 64 bits from main memory costs roughly 1,300 to 2,600 picojoules, against a few for a sum on the chip.

We score this paper 2 out of 10, which is low. It is a review with no new experiment. Two authors at one institution. No journal named, no code, no citations yet. Most of the input is vendor specification, and the authors say plainly that the ceilings are idealised upper bounds. So this business is a bet on the shape of the rule holding, and your own measurements have to carry the weight.

Who has this problem

Picture a 25 person app company in Warsaw. They make a notes app for field engineers. It works offline in basements and on roofs.

Leadership has decided this year is the AI year. The list has nine features on it. Summarise a site visit. Draft the customer email. Answer questions about the manual. All on the phone, because the client will not send site notes to a cloud.

Two engineers built a demo on their own new handsets. It felt fine. Then a tester on a three year old phone said the summary took forever, and the battery fell off a cliff.

Now the room is arguing. Nobody in it can say which of the nine features will work on the phones their users actually hold. So they argue from opinion, and they will build the wrong three.

What you would sell them

Sell a feature-fit audit. You take their nine planned features and their real device mix, and you hand back three lists. Ships now. Needs the model shrunk first. Cloud only.

The reasoning is on the page for each feature. How long is the answer this feature must write, in words. What is the speed ceiling on the phones in the bottom half of their user base. How often does the feature run, because a feature that fires on every keystroke is a battery problem before it is a speed problem.

On Monday morning the product manager has one page that ends the argument. Three of the nine get built. Two get a smaller model. Four go to the cloud or get dropped.

What to charge

Test it at 3,000 dollars for one audit of up to nine features on one app. That is a hypothesis and you should say so out loud.

Set it against building the wrong feature. One engineer for one month, on something that gets pulled after testing, is what the audit is competing with. You know what a month of your own engineering time costs. Put that number beside 3,000 dollars.

Then test a second version at a lower price for a single feature, because a team that will not risk 3,000 dollars may risk a tenth of that to settle one argument. Find out which door they walk through.

How you would build the first version

Do the sum, then check the sum on real handsets. That second half is the product.

Buy or borrow five phones across the range your clients care about. Old, middling and new. Run one small model on each. Measure tokens a second and battery drain for a fixed job. Then compare what you measured against the ceiling the sum predicts. Write down the gap.

That gap is your credibility. The paper’s ceilings assume one user, count only the model weights, and ignore the growing chat memory, the attention maths and scheduling. Real speeds come in lower. A buyer who sees you report your own measured gap will trust the rest of the page.

Do not build a dashboard. Do not build a device farm. Do not build an SDK. Use a spreadsheet, five phones and a stopwatch, and write the report by hand.

The one-week test

Day one. Set up five phones. Run the same model and the same job on each. Record tokens a second and battery drain.
Day two. Work out the ceiling for each phone from its published memory bandwidth and the model size. Put your measured number beside it. This table is your sample report.
Day three. Find 20 app teams who have announced an on-device AI feature or asked about one in public. Find the product manager, not the inbox.
Day four. Send the table with one question. Which of your planned AI features have you tested on your slowest supported phone? Offer the audit at 3,000 dollars and say you are testing that price.
Day five. Count signed orders. One paid audit from 20 teams is a yes. Zero, with replies saying they will just call the cloud, is a no.

What would kill this

The evidence is thin. This is a review, not an experiment. Two authors at one institution, no journal, no code. The ceilings are their own sums on published vendor figures. If you sell the ceiling as a promise, the first client who measures something different will be right and you will be wrong.

The ceiling is not the speed. The authors say these are upper bounds. They assume one user at a time and count only the model weights. They leave out the chat memory, which grows with the conversation. In their own example, a 175 billion parameter model at a very long context carries 618 gigabytes of chat memory against 350 gigabytes of weights. Quote a ceiling as a ceiling, never as a forecast.

Teams may just use the cloud. If your audit says cloud only nine times out of nine, nobody needs a second audit. Your market is teams with a real reason to stay on the device, such as private site notes or no signal, and that market may be smaller than it looks.

Every answer ages. New phones and smaller models change the result. An audit written today may be wrong next year. Either sell it again each year or accept that each client buys once.

Big teams already do this. Firms with their own performance engineers will not pay you. Sell to the ones with 25 people and nine ideas.

MEASURE THE PHONE YOUR USERS ACTUALLY HOLD. That is the whole service, and it stays true even if the paper’s exact numbers do not.

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.