Here's a question about death and probability, done first by Cory Doctorow. Suppose one out of every million people is a terrorist (if anything, an overestimate), and you've got a machine that can determine whether someone is a terrorist with 99.9 percent accuracy. You've used the machine on your buddy Jeff Smith, and it gives a positive result. What are the odds Jeff is a terrorist?
Try to figure it out, or at least guess, before you read on.
Here's the answer: a 0.1 percent chance — which is to say, the 99.9 percent accurate test will give you the wrong answer 99.9 percent of the time. Seems low, doesn't it?* This is the false positive paradox, and it completely blows up any possible security justification for the NSA's dragnet surveillance of our phone calls and emails.
The issue is stated simply and elegantly by Doctorow: "When you try to find something really rare, your test's accuracy has to match the rarity of the thing you're looking for." If it does not, then the number of false positives will completely bury the signal in irrelevant garbage.
And the numbers I used beforehand were not some crazy extrapolation. If anything, they were far too generous to the likes of the NSA. Their procedure for identifying terror suspects is not anything like a pushbutton machine, and is almost certainly less than 99.9 percent accurate. Instead, it's a colossal hodgepodge that has yet to produce any tangible successes.
Worse, there are no simple follow-up tests that might confirm the result. Instead, agents have to be dispatched to undertake a lengthy investigation, taking weeks or even months. As Bruce Schneier points out in his new book, this exact problem has bedeviled terrorism investigators:
In the years after 9/11, the NSA passed to the FBI thousands of tips per month; every one of them turned out to be a false alarm. The cost was enormous, and ended up frustrating the FBI agents who were obligated to investigate all the tips. We also saw this with the Suspicious Activity Reports —or SAR — database: tens of thousands of reports, and no actual results. And all the telephone metadata the NSA collected led to just one success: the conviction of a taxi driver who sent $8,500 to a Somali group that posed no direct threat to the U.S. — and that was probably trumped up so the NSA would have better talking points in front of Congress. [Data and Goliath]
Indeed, it's arguable that an obsessive focus on dragnet surveillance is actually a distraction from more effective investigative techniques, because even moderately competent terrorists will avoid electronic communication altogether. Bin Laden was suspicious of even encrypted email years before the Snowden leaks, but especially today, one would have to be grossly misinformed to express sympathy for terrorism online. This might explain why the FBI has spent so much time of late baiting utterly hapless chumps or the mentally ill into taking fake weapons and explosives they never would have been able to get on their own.
At any rate, as I've argued before, simple bureaucratic competence and bog-standard detective work are vastly underrated compared to piling up gigantic quantities of irrelevant data. But the false positive problem ought to be the final nail in the dragnet coffin. Unless terrorism becomes thousands of times more common than it is today, such broad techniques will be utterly useless against real terrorism.
*Out of every million people, 1 will be a terrorist, and 1000 (0.1 percent of 1 million) will be false positives. Therefore, Jeff's probability is 1/(1000+1) = 0.001, or 0.1 percent.
Correction: An earlier version of this article incorrectly stated Jeff's probability as 0.01 percent.
