Sell a pre-install check for the AI skills your staff download

The finding this rests on

Tencent Zhuque Lab released a free tool called AI-Infra-Guard. It checks an AI agent at four layers: the servers it runs on, its add-on tools, the way it behaves in conversation, and the model itself. The code is open source.

One part of it has a real score. The team gathered 62,652 skills from the main skill stores and boiled them down to 5,520 test cases across nine kinds of threat. On that set the skill scanner reached a loose F1 of 0.9848. Recall was 0.9974, so almost no bad skill slipped past. Precision was 0.9725.

One number matters more than the headline. The team ran the same rules nine times and changed only the model inside. The false alarm rate moved from 0.0120 to 0.1867. Therefore the model you put inside the scanner drives how often it cries wolf. Read the full write-up at A free tool checks AI agent add-ons for hidden attacks.

We score that paper 4 out of 10. It is a technical report, not a reviewed paper. Four of its five parts have no measured score at all, and the team scored its own scanner on its own benchmark. The benchmark and the code are public, which is the strongest point in its favour. So this idea is a bet that a scan that looks good on that test set also works on skills in the wild.

Who has this problem

Picture the IT lead at a 200-person company that sells insurance software. Her job is boring and important. She decides what staff may install.

Six months ago she knew every piece of software in the building. Then the engineers started adding skills and MCP servers to their agents. A skill is a small package that gives an agent new powers. One can read files, run commands, or open web pages.

Nobody asks her first. A developer finds a skill in a store on Tuesday, installs it on his laptop, and it now has the same reach he does. She has no list of what is installed, no way to check one, and no idea who wrote any of it.

She has seen this shape before with browser add-ons. She knows how that ended.

What you would sell them

Sell a gate and a list. Staff send you a link to any skill or MCP server before they install it, and within one hour they get back one word. Green, amber or red, with one line saying why.

You also keep the list. Every approved skill sits on one page the IT lead can read, with the date it was checked and who asked for it. Once a week you re-check everything on that list, because a skill can change after you approve it.

On Monday morning she opens one page and knows what her company is running.

What to charge

Test it at 400 dollars a month for one company, covering up to fifty skills. Test a second price of 25 dollars for a single check, for people who will not commit to a month. Both are hypotheses. A payment is the only proof.

Here is the reasoning. Do not compare your fee to the scanner, because the scanner is free and your customer can download it. Compare it to the alternatives she has now. Ask her what she already pays per month for one seat of any security tool the company buys, then price below it. Ask her what one rebuilt laptop costs the company in time. Your job is to be cheaper than one bad Tuesday.

How you would build the first version

Run the free tool as it comes. Do not write a scanner.

Do one piece of homework first. The benchmark is public, so run it yourself before you sell anything. Try several models inside the scanner and write down the false alarm rate for each. The study found that number swinging from 0.0120 to 0.1867 on the same rules, so this choice is your product quality. Pick the model that raises fewer false alarms, then measure what one scan costs you to run, because the report does not say and each scan calls a paid model many times.

The rest is a shared inbox and a spreadsheet. Requests come in by email. Verdicts go out by email. The approved list is one page with four columns.

Do not build a browser plug-in. Do not build a tool that blocks installs on a laptop. Do not build your own benchmark. Do not promise to check anything in under a minute yet.

The one-week test

1. Day one. Install the free tool. Run the public benchmark with three different models inside and record the false alarm rate for each.
2. Day two. Scan twenty real skills from a public store. Time each scan and write the verdict in plain words a non-engineer would understand.
3. Day three. Ask ten IT leads one question. How many agent skills are installed in your company right now? Count how many cannot answer.
4. Day four. Ask three of them for the skill list from ten laptops. Scan it free and send back the one page list.
5. Day five. Ask all ten for 400 dollars for the first month. Send an invoice with a start date.

By Friday the answer is yes or no. Yes means at least one company has paid or signed. No means they all want to see it again after the audit. After the audit is a no.

What would kill this

• Most of the tool is unproven. Only the skill scanner has a measured score. The server scanner, the MCP auditor, the agent red teamer and the jailbreak tester are described by how many rules they hold, not by how well they work. The authors say plainly that rule counts describe scope, not effectiveness. You are selling on one number out of five.
• The score comes from the team that built the tool. They gathered the skills, built the benchmark, ran the leaderboard and scored their own scanner. The data is public, which helps. But no outside group has scanned skills in the wild and reported what the tool missed, and the authors list that as future work.
• Green is a promise you cannot fully keep. The authors warn that the plug-in checker tests the shape of a plug-in, not its safety, so a bad plug-in can still run code. Never write the word safe in a report. Write what you checked and what you did not.
• Your margin is unknown. Each scan calls a paid model many times, and the report does not say what that costs. Measure it in week one. If one check costs more than a few dollars, a 400 dollar month with heavy users loses money.
• The software is free. Your customer can download the same tool tonight. You are selling the queue, the record and somebody to answer within an hour. If that is not worth paying for, there is no business here.
• A slow gate gets walked around. If your answer takes a day, the developer installs the skill on Tuesday anyway and tells you on Friday. The whole product dies at the speed of your reply.

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.