The NSA’s Gigantic Haystack

Big Data Meets Big Surveillance

NSA spying large

For decades, detectives investigating crimes committed in the U.S. have been able to reach out to phone companies and find out what calls were made from someone’s phone. Under a Supreme Court ruling from 1979, no warrant was needed unless the police also wanted to listen in. Today, as part of an effort to uncover terrorist plots before they are put into action, the government tracks all of the calls made by almost every American. Revelations about National Security Agency actions from Edward Snowden, the former NSA contractor who stole an estimated 1.7 million secret files, included a program known as Prism, in which the U.S. compels companies through court orders to turn over text and e-mails of users. Congress, the courts, President Barack Obama and the public are now wrestling with the question of how much information the government should be allowed to collect. How big does a pile of mundane data about ordinary citizens have to be before the people keeping us safe turn into Big Brother?

The Situation

In February 2015, Obama announced some restraints on NSA spying, though they didn’t affect the NSA’s bulk collection of data. In May, the first appeals court to rule on phone record collection found the program was illegal because it went beyond authority granted by the Patriot Act, a law passed after 9/11 that gave the government broad new investigative powers. It didn’t order the program blocked, since portions of the Patriot Act expired June 1 — including the section the government used to justify collecting bulk phone records — and its fate was being weighed in Congress. The House and Senate then voted to end this bulk collection and President Obama signed the bill into law. The NSA is continuing to collect bulk records while a new system is put in place in which phone companies store the data and investigators request it when needed.

Source: U.S. Courts wiretap report
Source: U.S. Courts wiretap report

The Background

Every step forward in communication — telegraph, telephone, e-mail — has been embraced by police and intelligence agencies as a new way of listening in, sometimes within the law, sometimes not. What’s different is the shift from targeting individuals to mass collection. In the 1990s, the FBI and others started vacuuming up data from phone calls and e-mails. Snowden’s leaks showed how the combination of powerful data analysis tools, a loosened legal framework in the wake of the 9/11 attacks and the widespread adoption of smartphones laid the groundwork for data collection on a previously unimaginable scale. The NSA and the FBI still need a warrant to listen to the content of calls, but the metadata they collect about the calls can let them know who you talked to, when, from where and for how long, along with who those people call, and who their contacts call.

The Argument

An advisory panel appointed by Obama concluded that the phone records program “was not essential to preventing attacks” and a privacy board created by Congress reached a similar conclusion. In October, the European Union’s top court ruled that a 15-year-old agreement that allowed American companies to move commercial data back home was invalid, because it no longer gave European citizens adequate protection against mass spying by U.S. security agencies. Officials who oversee the spy programs say they need to compile a giant haystack of data to find needles quickly; they dismiss privacy concerns as overblown. Director of National Intelligence James Clapper told Congress that snooping should be measured by a “peace-of-mind metric.” Supporters of the bulk collection say it’s justified by the 1979 Supreme Court decision, which held that since everyone knew the phone company kept records of their calls there was no expectation of privacy. Critics challenge that interpretation, and say that safeguards that may have been adequate in the days of dial telephones are meaningless in the age of smartphones. Revelations of the agency’s use of web bugs raised questions about how much insecurity the NSA was willing to create to keep Americans secure.

The Reference Shelf

  • Barack Obama’s August 2007 speech at Washington’s Wilson Center pledging that as president there would be “no more illegal wiretapping of American citizens.”
  • The bill limiting bulk collection of data by the NSA passed by the House of Representatives in May.
  • American Civil Liberties Union report: “How the NSA’s Surveillance Procedures Threaten Americans’ Privacy.”
  • Wired magazine’s 2012 article on the NSA’s metadata collection center in Utah.

To receive a free monthly QuickTake newsletter, sign up at

First Published Jan. 23, 2014

To contact the writer of this QuickTake:

Chris Strohm in Washington at

To contact the editor responsible for this QuickTake:

Anne Cronin at