Climate & energy tech/United States
They released metered methane in Arizona for a year and did not tell the satellites how much
A year-long blind test released metered methane under twelve satellites. Purpose-built instruments did well; AI methods produced 79 percent of the false alarms.
A year-long single-blind controlled release test in Arizona involved 746 events - 481 methane releases of 9.5 to 1,183.7 kilograms per hour and 265 blank controls - observed by 12 satellites. Fifteen analysis teams from nine countries produced over 2,250 estimates. Purpose-built instruments achieved 92 to 100 percent true positive rates with 44 percent mean absolute percentage error; AI-based methods produced 79 percent of all false positives.
What happened
Methane satellites have become central to climate enforcement. Operators publish plume detections, regulators cite them, companies are asked to explain them. Almost none of it has been independently verified, because verifying it requires knowing exactly how much gas came out of a specific point at a specific moment – and nobody tells the satellites.
A team led from the Department of Energy Science and Engineering at Stanford University, with Climate and Space Sciences and Engineering at the University of Michigan and the Energy and Biosciences Institute at the University of California-Berkeley, spent a year doing exactly that.
They released metered quantities of methane in Arizona, timed to satellite overpasses, and did not tell the analysts what they had released. Fifteen teams from nine countries filed over 2,250 estimates.
The test
746 events: 481 methane releases ranging from 9.5 to 1,183.7 kilograms per hour, and 265 blank controls where nothing was released.
The blanks are what make this a real test. A detection system evaluated only on releases can score perfectly by claiming to see methane every time. Including blanks measures the false positive rate, and for an instrument used in enforcement that is the error that costs someone money.
Releases coincided with overpasses from 12 government, commercial and non-profit satellites. The test included the first evaluations of EMIT, Tanager-1 from Carbon Mapper, and GHOSt.
Single-blind means the analysis teams knew the observation windows and not the release quantities. Fifteen teams from nine countries participated, making this, to the authors’ knowledge, the largest controlled test of its kind.
The numbers
| Instrument class | Detection performance | Quantification error |
|---|---|---|
| Built-for-purpose methane instruments | 92-100% true positive rate, 0-11% false positive rate | slope 0.83-1.00, 44% MAPE |
| Repurposed instruments | 62-88% detection rate | 46% MAPE |
| Broad land scanners | poor at tested rates | – |
| AI-based methods | 79% of all false positives originated here | |
| Cloud cover above 20% | degraded performance across all systems | |
| Release range | 9.5 to 1,183.7 kg per hour | |
| Events / estimates | 746 events, over 2,250 estimates | |
A slope of 0.83 to 1.00 means the best instruments’ estimates scale roughly correctly with the true release rate. A 44 percent mean absolute percentage error means individual estimates are, on average, wrong by nearly half.
The result
Performance was heterogeneous, and the heterogeneity is the finding.
Instruments designed specifically to target methane detected 92 to 100 percent of releases with false positive rates between 0 and 11 percent, and quantified with slopes of 0.83 to 1.00 and 44 percent mean absolute percentage error.
Instruments repurposed for methane sensing performed more variably, detecting 62 to 88 percent with 46 percent mean absolute percentage error – still, the authors note, demonstrating significant skill.
Broad land scanners detected poorly at the tested rates, though they have consistent global coverage and historical archives, which the authors say motivates further work.
AI-based methods showed promise with a caveat the paper states plainly: 79 percent of all false positives emerged from these methods.
Ground-measured cloud cover above 20 percent degraded performance across all systems.
The authors conclude that satellite-based measurements can achieve accurate detection and quantification of methane point sources when the source location is approximately known.
How this compares to what came before
Previous controlled release tests have been smaller, typically covering one or two instruments over a few weeks, and some have not been blind. Those tests established that controlled release evaluation is possible.
This is 746 events over a year with twelve satellites and fifteen independent analysis teams, including three instruments evaluated for the first time. The scale is what allows the comparison across instrument classes, which is the thing operators and regulators actually need.
The AI false positive finding has no precedent because there has been no independent test large enough to measure it. Machine learning plume detection has been adopted rapidly across the sector on the strength of self-reported performance.
What it means
For regulators and anyone acting on satellite methane data, the instrument class now matters formally. A detection from a purpose-built methane instrument and a detection from a repurposed sensor are different kinds of evidence, and this test quantifies the difference.
The 44 percent quantification error is the number to carry into any policy conversation. These systems are good at saying a leak exists and coarse at saying how large it is, which means they support enforcement triggers better than they support emissions accounting.
The AI result deserves to change practice. Machine learning methods generating four in five false positives is not an argument against using them; it is an argument for human verification of AI-flagged detections before anyone is accused of anything.
And the caveat in the authors’ own conclusion is the one most likely to be dropped in reporting. These results hold when the source location is approximately known. Searching a large area for unknown leaks is a harder problem that this test did not evaluate.
Business ideas from this paper
Verification services for AI-flagged detections. If AI methods produce 79 percent of false positives, the market need is a verification layer between detection and action. That is a service with regulatory demand behind it.
Instrument-class-aware data products. Aggregators combining multiple satellite feeds should weight them by measured performance. This test provides the weights and nobody is using them yet.
Controlled release testing as a service. The methodology is now established and reusable. Running periodic blind tests for new instruments and new algorithms is infrastructure the sector needs permanently.
Cloud-aware coverage planning. Performance degrades above 20 percent cloud cover across all systems. A product that tells an operator which regions and seasons are effectively unmonitorable would change how monitoring programmes are designed.
What could be wrong with it
The releases were at known locations in Arizona, and the authors state their conclusion holds when the source location is approximately known. Wide-area surveying for unknown sources is the harder use case and was not tested.
Release rates ran from 9.5 to 1,183.7 kilograms per hour. Smaller leaks, which are collectively significant, were not covered.
One site, one terrain type, one set of atmospheric conditions. Arizona desert is favourable for optical detection, and performance over vegetation, wetland or snow would differ.
Even the best instruments carried 44 percent mean absolute percentage error, so quantification remains coarse.
Code and data availability were not stated in the record we checked, which for a benchmarking study is a notable gap.
How sure can you be?
We score this 9 out of 10.
It is a peer-reviewed paper in Nature reporting a year-long single-blind controlled release test with 746 events including 265 blank controls, evaluated by fifteen independent analysis teams from nine countries across twelve satellites, with named institutions at Stanford University, the University of Michigan and the University of California-Berkeley.
It loses a point because we could not confirm that the data are publicly deposited.
The independent blind evaluation is as strong as evidence gets in remote sensing. This is the design every sensing technology should be subjected to and almost none are.
Do this today
If you act on satellite methane data, find out which instrument class produced the detection. The difference between 92-100 percent and 62-88 percent detection is the difference between evidence and a lead.
If you use AI-based plume detection, add a verification step. Four in five false positives across this test came from these methods, and acting on an unverified one has consequences.
If you design monitoring programmes, build the cloud cover limit into your coverage claims. Above 20 percent cloud, every system tested degraded, and a monitoring programme that does not account for that is overstating its coverage.
Source: Evaluating space-based methane detection through a one-year blind test, Nature, October 2026. PMID 42875571 · doi.org ().
Just Out Tech explains new research in plain language. This article was drafted with AI assistance and checked by a human against the original source.
- Purpose-built methane instruments: 92-100 percent true positive rate, 0-11 percent false positive rate, 44 percent mean absolute percentage error.
- Repurposed instruments: 62-88 percent detection, 46 percent mean absolute percentage error.
- AI-based methods produced 79 percent of all false positives despite showing promise.
- Cloud cover above 20 percent degraded performance across every system tested.
Twelve government, commercial and non-profit satellites already in orbit were tested against metered releases over a full year, by fifteen analysis teams in nine countries.
Questions people ask
What is a single-blind controlled release?
The organisers release known, metered quantities of methane at known times. The analysis teams know when overpasses happened but not whether gas was released or how much. It is the gold standard for evaluating a detection system.
Why include blank controls?
265 of the 746 events were blanks - no release. Without them you cannot measure the false positive rate, which is the error that matters most for an instrument used in regulation or enforcement.
What does 44 percent mean absolute percentage error mean?
That even the best instruments, on average, got the release rate wrong by 44 percent. They detect well and quantify coarsely.
Why is the AI finding important?
AI-based methods showed promise but generated 79 percent of all false positives. For a technology being proposed for regulatory monitoring, a false positive means accusing an operator of a leak that did not happen.