Opinion

The simple math problem that blows apart the NSA's surveillance justifications

Here's why the NSA dragnet won't work

Here's a question about death and probability, done first by Cory Doctorow. Suppose one out of every million people is a terrorist (if anything, an overestimate), and you've got a machine that can determine whether someone is a terrorist with 99.9 percent accuracy. You've used the machine on your buddy Jeff Smith, and it gives a positive result. What are the odds Jeff is a terrorist?

Try to figure it out, or at least guess, before you read on.

Here's the answer: a 0.1 percent chance — which is to say, the 99.9 percent accurate test will give you the wrong answer 99.9 percent of the time. Seems low, doesn't it?* This is the false positive paradox, and it completely blows up any possible security justification for the NSA's dragnet surveillance of our phone calls and emails.

The issue is stated simply and elegantly by Doctorow: "When you try to find something really rare, your test's accuracy has to match the rarity of the thing you're looking for." If it does not, then the number of false positives will completely bury the signal in irrelevant garbage.

And the numbers I used beforehand were not some crazy extrapolation. If anything, they were far too generous to the likes of the NSA. Their procedure for identifying terror suspects is not anything like a pushbutton machine, and is almost certainly less than 99.9 percent accurate. Instead, it's a colossal hodgepodge that has yet to produce any tangible successes.

Worse, there are no simple follow-up tests that might confirm the result. Instead, agents have to be dispatched to undertake a lengthy investigation, taking weeks or even months. As Bruce Schneier points out in his new book, this exact problem has bedeviled terrorism investigators:

In the years after 9/11, the NSA passed to the FBI thousands of tips per month; every one of them turned out to be a false alarm. The cost was enormous, and ended up frustrating the FBI agents who were obligated to investigate all the tips. We also saw this with the Suspicious Activity Reports —or SAR — database: tens of thousands of reports, and no actual results. And all the telephone metadata the NSA collected led to just one success: the conviction of a taxi driver who sent $8,500 to a Somali group that posed no direct threat to the U.S. — and that was probably trumped up so the NSA would have better talking points in front of Congress. [Data and Goliath]

Indeed, it's arguable that an obsessive focus on dragnet surveillance is actually a distraction from more effective investigative techniques, because even moderately competent terrorists will avoid electronic communication altogether. Bin Laden was suspicious of even encrypted email years before the Snowden leaks, but especially today, one would have to be grossly misinformed to express sympathy for terrorism online. This might explain why the FBI has spent so much time of late baiting utterly hapless chumps or the mentally ill into taking fake weapons and explosives they never would have been able to get on their own.

At any rate, as I've argued before, simple bureaucratic competence and bog-standard detective work are vastly underrated compared to piling up gigantic quantities of irrelevant data. But the false positive problem ought to be the final nail in the dragnet coffin. Unless terrorism becomes thousands of times more common than it is today, such broad techniques will be utterly useless against real terrorism.

*Out of every million people, 1 will be a terrorist, and 1000 (0.1 percent of 1 million) will be false positives. Therefore, Jeff's probability is 1/(1000+1) = 0.001, or 0.1 percent.

Correction: An earlier version of this article incorrectly stated Jeff's probability as 0.01 percent.

More From...

Picture of Ryan CooperRyan Cooper
Read All
Losing the vaccinated majority?
President Biden.
Opinion

Losing the vaccinated majority?

Microsoft's Activision Blizzard bailout
Bobby Kotick.
Opinion

Microsoft's Activision Blizzard bailout

America's long record of judicial despotism
Roger B. Taney.
Opinion

America's long record of judicial despotism

Chile's bold political experiment is a lesson for Americans
Gabriel Boric.
Opinion

Chile's bold political experiment is a lesson for Americans

Recommended

Texas Gov. Greg Abbott takes fire over National Guard border deployment
Texas National Guard at Mexico border
'mall cops on the border'

Texas Gov. Greg Abbott takes fire over National Guard border deployment

10 things you need to know today: January 20, 2022
Biden faces the press on Jan. 19, 2022
Daily briefing

10 things you need to know today: January 20, 2022

The futility of Biden's 1st year
President Biden.
Picture of David FarisDavid Faris

The futility of Biden's 1st year

Cory Booker, Tim Scott spar over comparing voting restrictions to Jim Crow laws
Sen. Cory Booker (D-N.J.).
voting rights fight

Cory Booker, Tim Scott spar over comparing voting restrictions to Jim Crow laws

Most Popular

California deputy DA opposed to vaccine mandates dies of COVID-19
Kelly Ernby.
covid-19

California deputy DA opposed to vaccine mandates dies of COVID-19

Joe Biden meets the press
President Biden.
Picture of Joel MathisJoel Mathis

Joe Biden meets the press

Fans raise thousands of dollars for animal shelters as part of the Betty White Challenge
Betty White holds an adorable puppy.
thank you for being a friend

Fans raise thousands of dollars for animal shelters as part of the Betty White Challenge