This is why you can't trust the NSA. Ever.

The notion that the National Security Agency could police its own internet dragnet program with minimal oversight from a secret court has long drawn scoffs from observers. Now it appears that skepticism was completely justified, following the release of a bunch of documents on the program earlier this month by the office of Director of National Intelligence James Clapper (ODNI), which came in response to a Freedom of Information Act request filed by the Electronic Privacy Information Center.

Exhibit A is a comprehensive end-to-end report that the NSA conducted in late summer or early fall of 2009, which focused on the work the agency did in metadata collection and analysis to try and identify people emailing terrorist suspects.

The report described a number of violations that the NSA had cleaned up since the beginning of that year — including using automatic alerts that had not been authorized and giving the FBI and CIA direct access to a database of query results. It concluded the internet dragnet was in pretty good shape. "NSA has taken significant steps designed to eliminate the possibility of any future compliance issues," the last line of the report read, "and to ensure that mechanisms are in place to detect and respond quickly if any were to occur."

Subscribe to The Week

Escape your echo chamber. Get the facts behind the news, plus analysis from multiple perspectives.

SUBSCRIBE & SAVE

Sign up for The Week's Free Newsletters

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

Sign up

But just weeks later, the Department of Justice informed the FISA Court, which oversees the NSA program, that the NSA had been collecting impermissible categories of data — potentially including content — for all five years of the program's existence.

The Justice Department said the violation had been discovered by NSA's general counsel, which since a previous violation in 2004 had been required to do two spot checks of the data quarterly to make sure NSA had complied with FISC orders. But the general counsel had found the problem only after years of not finding it. The Justice Department later told the court that "virtually every" internet dragnet record "contains some metadata that was authorized for collection and some metadata that was not authorized for collection." In other words, in the more than 25 checks the NSA's general counsel should have done from 2004 to 2009, it never once found this unauthorized data.

The following year, Judge John Bates, then head of FISC, emphasized that the NSA had missed the unauthorized data in its comprehensive report. He noted "the extraordinary fact that NSA's end-to-end review overlooked unauthorized acquisitions that were documented in virtually every record of what was acquired." Bates went on, "[I]t must be added that those responsible for conducting oversight at NSA failed to do so effectively."

Nevertheless, in the very same document, Bates would go on to authorize restarting the program (his colleague, Judge Reggie Walton, had shut it down after learning of the illegal collection in late 2009). Not only that: Bates's reauthorization permitted the NSA to collect all the data it had been unauthorized to collect before; expanded the number of NSA analysts who could access the data to pretty much anyone with training; unmoored the collection from specific switches more likely to carry terrorist traffic; and expanded the volume of collection by 11 to 24 times.

In other words, Bates decided it was a good idea to let those who, in his judgment, failed to effectively conduct oversight at the NSA to dramatically expand the program.

While a sketchy outline of this story was revealed when the government first released Bates's 2010 reauthorization memo last year, a clearer picture emerged following ODNI's document dump.

Those documents also show how lawyers from the Justice Department secretly told FISC Judge Colleen Kollar-Kotelly in 2004 that they couldn't get Congress to pass a law to expand the executive's spying authority — which they admitted was what the White House normally did when it came across a law it found too restrictive — because "seeking legislation would inevitably compromise the secrecy of the collection program the government wishes to undertake."

The documents show that when Attorney General Alberto Gonzales briefed the Senate Intelligence Committee in 2005 on programs authorized by the internet dragnet law, he made no mention of what the NSA was doing under it. They show how Judge Walton correctly guessed, in early 2009, that he might find the same violations with the internet dragnet as the Justice Department had previously disclosed about the NSA's phone-tapping program.

And of course, they show that roughly six months of close review of the internet dragnet program did not lead the NSA to discover — or if it did discover, to admit — that it had been illegally collecting data within the U.S. for five whole years. It took something else to get NSA to admit to that.

Clapper's office maintains that these documents demonstrate "the oversight regime of internal checks over the program." Perhaps, though they reflect favorably only on Walton's decision to shut the program down in 2009.

But there's a lot Clapper's office isn't saying. First, his office is hiding almost all the dates on these documents (it took matching these with many other public documents to come up with the estimates in this article). Perhaps that's to shield the government from liability for this illegal spying.

Also, ODNI claims that the FISC-authorized internet dragnet has been shut down. "As previously stated, this internet communications metadata bulk collection program has been discontinued." But during precisely the same weeks when NSA's general counsel was busy not finding the illegal data in virtually every internet dragnet record, NSA piloted a new program to permit its analysts to do the same kind of analysis on the metadata of U.S. persons collected under an executive order (Executive Order 12333). NSA expanded the program to all of NSA in early 2011, before NSA shut down the internet dragnet program.

That means there is a related dragnet program out there with nowhere near the level of oversight as the old one — the one that managed to compile five years of serious violations that the NSA never detected.