More than half of MCP security alerts are false alarms

What happened

Think about a smoke alarm that screams every time you make toast. Therefore after a week you stop looking up. Then one day there is a real fire, and nobody moves.

AI agents plug into small programs called MCP servers, and those servers can read your files, run commands and send data out. Several free tools claim to scan them and tell you which ones are dangerous. Those tools say almost every server is risky.

A team in Shanghai decided to check the checkers. They gathered tens of thousands of real MCP servers, got them running, and pointed eight popular scanners at them. Then they read the warnings by hand. However, most of the warnings did not hold up.

The test

The paper is “Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability”. Pei Chen, Baichao An, Mengying Wu and colleagues at Fudan University and the Shanghai Innovation Institute wrote it. It went on arXiv on 13 July 2026.

The hard part came first, because code sitting in a public repository is not a running service. Most of these projects will not start on their own. The team found that 78.6 percent of servers ship no Dockerfile at all. Even among projects with a full set of files, only 19.6 percent ran without changes.

So they built a set of AI helper programs that read each project, guess what it needs, try to build it, read the error, and try again. That got 57.7 percent of projects running. The result is a collection they call MCPZoo. It holds 64,611 unique server projects, 113,927 counting copies, and 37,288 that can be poked at while running.

They then ran eight scanners over the running servers. Every scanner was given the same controlled setup and the same local model to think with. Finally, two people sat down and checked 100 flagged servers by hand, reading the code and the evidence. The two reviewers agreed strongly, and a third settled any argument.

The result

The scanners flagged 96.89 percent of running servers as risky, and that sounds like a burning building.

The hand check tells another story. On average, only 45.53 percent of the sampled alerts were real. The worst scanner was right 10.40 percent of the time. The best was right 96.88 percent of the time, but it hardly flagged anything.

The scanners also miss real bugs. The team pulled 10 published CVEs that affect 38 servers in their collection. The scanners spotted 24.17 percent of those known problems, and several scanners found none at all.

And the scanners do not agree with each other. Measure how much any two of them overlap and you get 15.66 percent on average. Not one server was flagged by all eight. On specific problems it is worse. For prompt injection the overlap is 5.98 percent, and for data leakage it is 3.36 percent.

What it means

A warning is only useful if it means something. Here is a real example from the paper. One scanner reported a leaked password because a field was named “token”. When the team ran the server, that field was an empty box waiting for input. No secret, no leak, just a word.

That is the pattern. Scanners look at names, descriptions and shapes, and they then guess at danger. However, they rarely check whether an attacker could actually do the thing. IF EVERYTHING IS FLAGGED, NOTHING IS FLAGGED.

There is a second lesson hiding in the numbers. The MCP world looks bigger than it is, because one template appears 537 times. The team also found hundreds of near-identical copies of a server template that requires no login. Therefore one weak design becomes hundreds of weak servers, and one scary looking scan report may be the same server counted many times.

One more number is worth keeping. Most servers offer fewer than 10 tools, but 37.6 percent of all tools can run commands, change files or send data out. The power is real, and it is the measuring that is broken.

Business ideas from this paper

1. A second-opinion scan. Run four scanners, then only report a server when two or more agree. The paper shows that agreement is rare, so agreement is worth something. Who buys it: teams that already run one scanner and do not trust its output. Price to test: 99 dollars a month. One-week test: take the last 50 alerts from three teams, rescore them, and ask each team how many alerts they can now safely ignore.
2. A copy detector for MCP servers. Take a server, compare its tools and text against known templates, and say “this is copy number 300 of a template with no login”. Who buys it: the person who approves new tools inside a company. Price to test: 39 dollars a month. One-week test: scan the servers of ten teams for free and count how many find a duplicate they did not know about.
3. A make-it-run service. Nearly four in five MCP projects ship no container file, so most cannot be tested at all. Sell a container plus a health check for any server a customer wants to use. Who buys it: companies that want to run a public MCP server in-house instead of trusting a hosted one. Price to test: 250 dollars per server, one payment. One-week test: offer it for five servers and see if two customers pay before you build anything.

How sure can you be?

The scale here is real, and so are the limits.

This is a preprint, and the paper names no journal and no conference.

The hand check covered 100 servers, not thousands. That is a sample, and precision figures for single scanners rest on smaller counts still. The CVE test is smaller again: 10 published bugs across 38 servers. The authors say so themselves and call a bigger ground truth an important direction for future work.

The authors also flag a bias in their own collection. MCPZoo leans toward projects that are easy to deploy, because servers that need a paid API key or special hardware never started. Those may not look like the hardest cases.

Results also depend on which scanner version was tested and which model it used inside. The team used one fixed local model for every scanner that needed one. A better model might change the scores.

Finally, the data is not fully open. There is a public page where you can look up a server, but the collection itself is promised under controlled access for academic use. So you cannot rerun this study yourself today.

What would settle it? A larger hand-checked set of confirmed bugs, checked by a group that did not build MCPZoo.

Do this today

If a scanner flags your MCP server, do not panic and do not ignore it. Open the code at the line it names and ask one question: could an outsider actually reach this?

Source: Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability, July 2026. arXiv:2607.11086 · arxiv.org (preprint · not yet peer reviewed).

Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.