what is a heat map explanation?

It is a coloured overlay that shows which parts of an image an AI model used for its answer. Doctors and auditors use it as a sanity check. This paper tested 67 setups of such maps on a skin image model.

why is a perfect steadiness score a bad sign?

Steadiness here means the map does not change much when the input is nudged. A map that never changes gets a perfect score without saying anything. In this paper one setup scored 1.000 for steadiness and 0.0502 for truthfulness.

did the team run this on a real device?

No. They wrote down a profile of a Cortex-M7 chip with 512 kB of fast memory and worked to that on paper. Timing was taken on a normal computer. The authors state that physical microcontroller deployment was outside the scope of the study.

how accurate was the skin model itself?

It reached 66.77 percent on 2,004 held-out photos, with a test loss of 1.0641. That is a weak model. The paper is about choosing explanations, not about building an accurate skin classifier.

A perfect steadiness score hid a useless heat map

What happened

A nurse holds a small device up to a mole on your arm. The screen says the mole is probably harmless. It also paints a red blob over the part of the photo it looked at. Should she believe the blob?

That red blob is called a heat map. It is meant to show the reason behind an answer. Doctors want one before they act on a machine. But heat maps cost memory and time, and a tiny medical device has very little of either.

A team at the University of Hull tried to pick the best heat map for such a device. Three things pull against each other. How truthful the map is. How steady it is. How much room it needs. The team built a way to weigh all three at once.

The test

The paper is called “Human-Centered Explainable AI for TinyML Edge Devices: A Pareto-Based Selection Framework with LLM-Guided Design”. It was written by Zeinab Dehghani, Dhavalkumar Thakker, Koorosh Aslansefat and colleagues. All seven authors work at the School of Digital and Physical Sciences at the University of Hull in the United Kingdom. It went up on arXiv on 7 August 2026.

They used a public set of skin photos called HAM10000. It holds 10,015 images across seven kinds of skin mark. They split it into 7,010 photos for training, 1,001 for tuning and 2,004 for the final test. The model was MobileNetV3-Small, a small image model.

Then they tested 67 different heat map setups. Two AI chat models, GPT-4.1 mini and Gemini 2.0 Flash, suggested which methods to try. Hard rules then threw out any method that could not fit the device. The rest were ranked. Finally GPT-4.1 mini picked from the short list it was allowed to touch.

No real device was used. The team wrote down a profile of a common chip, a Cortex-M7 with 512 kB of fast memory, and worked to that on paper. Timing was taken on a normal computer.

The result

The most useful finding is a warning. One setup scored a perfect 1.000 for steadiness. Its truthfulness score was 0.0502 out of 1.

Why? Because it barely changed. Feed it two different photos and the map looks about the same. A map that never moves is very steady. It is also close to useless. Steadiness alone rewards exactly the wrong thing.

The plain method won instead. Plain CAM matched the best truthfulness score of 0.9397 and had the lowest cost of the lot, 0.150. It survived every filter under all three sets of goals. The AI picked it for all three.

There is a second surprise. Smaller did not mean faster. One method shrank its working memory from 27.6 kB to 9.0 kB. But it added 0.203 milliseconds of work, far more than a method that kept the larger memory and added under 0.012 milliseconds.

One more number is worth knowing. The model itself scored 66.77 percent on the 2,004 test photos. So these heat maps explain a model that is wrong about a third of the time.

What it means

Teams pick tools by score. This paper shows how a score can be gamed by accident. ANY SINGLE SCORE CAN BE GAMED BY AN ANSWER THAT NEVER CHANGES. That is the line worth keeping.

You can check for this yourself in a minute. Take two different inputs. Look at the two explanations side by side. If they look the same, the explanation is telling you nothing about either one.

There is a money lesson too. The plainest and cheapest method beat the clever ones on this task. Before you pay for a fancier tool, test the free one that has been around for years. It may already sit on the best trade off.

And keep the order right. An explanation cannot fix a weak model. A pretty red blob over a coin flip is still a coin flip.

Business ideas from this paper

A small checker that runs a model on 20 different inputs and flags explanations that barely change between them. Who buys it: teams shipping AI in medical, safety or money products, where someone must sign off. A price to test: 50 dollars a month for one project. A one-week test: run it free on 20 open image models, post the ones that fail, and count how many teams ask you to check their own.
A memory budget sheet that says which AI features fit on a given microcontroller before anyone writes code. Who buys it: product teams putting AI into meters, monitors and farm sensors. A price to test: 300 dollars per board report. A one-week test: publish a free sheet for three popular boards, then count requests for a fourth.
A short training session that teaches clinic staff how to read an AI heat map, and when to ignore it. Who buys it: clinic and lab managers rolling out AI screening tools. A price to test: 600 dollars per session for up to twelve people. A one-week test: run one free session at a single clinic, then ask for three named referrals at the end.

How sure can you be?

Treat this as an early result. The authors call it a proof of concept and they are right to. They are also unusually clear about what is missing, which is a good sign.

Nothing ran on a real tiny device. The chip was a written profile, not hardware. Timing was measured on a normal computer, so the millisecond figures are comparisons and not device speeds. The memory cost was an estimate, not a measured energy figure.

No doctor looked at the heat maps. The team did not compare the maps against marked outlines of the skin marks. So nobody has yet shown that the winning map points where a dermatologist would point.

The work also covers one data set, one model and one task. The AI selection step was run once per goal set, and the authors say a future study should check how much the answer moves across repeated runs.

Three things would settle it. Real measurements on a real chip. Doctors rating the maps. A repeat on a second disease and a second model.

Do this today

Open your own model, run two clearly different inputs through it, and put the two explanations side by side. If you cannot tell them apart, stop trusting that explanation today.

Source: Human-Centered Explainable AI for TinyML Edge Devices: A Pareto-Based Selection Framework with LLM-Guided Design, August 2026. arXiv:2608.07091 · arxiv.org (preprint · not yet peer reviewed).

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.