The NSA’s Gigantic Haystack

Big Data Meets Big Surveillance

NSA spying large

For decades, detectives investigating crimes committed in the U.S. have been able to call up phone companies and find out what calls were made from someone’s phone. Under a Supreme Court ruling from 1979, no warrant was needed unless the police also wanted to listen in. Today, as part of an effort to uncover terrorist plots before they are put into action, the government tracks all of the calls made by almost every American, along with text messages and e-mails. Thanks to revelations about National Security Agency programs from Edward Snowden, the former NSA contractor who stole an estimated 1.7 million secret files, Congress, the courts, President Barack Obama and the public are wrestling with the question of how much information the government should be allowed to collect. How big does a pile of mundane data about ordinary citizens have to be before the people keeping us safe turn into Big Brother?

The Situation

By the end of 2013, a consensus had developed in Washington that the NSA program needed to be changed, but fights over how to change it are not likely to end soon. An advisory panel appointed by Obama concluded that the phone records program “was not essential to preventing attacks” and a privacy board created by Congress reached a similar conclusion. After months of negotiations, the House in May passed a bill that would require phone companies to store the data rather than the NSA, and would require the government to get an order from the Foreign Intelligence Surveillance Court, a secret panel of judges, before searching them. A group of technology companies, including Facebook, Google and Apple, opposed the bill because of what it called an “unacceptable loophole that could enable the bulk collection of Internet users’ data.” The bill then headed to the Senate. Separately, reports emerged that the NSA had learned of the so-called Heartbleed bug, a flaw in Internet security software, two years ago — but had chosen to exploit the weakness to collect passwords and other information rather than report it.

The Background

While government surveillance goes way back — remember David watching Bathsheba? — it took the harnessing of electrons to give the practice real reach. Every step forward in communication — telegraph, telephone, e-mail — has been embraced by police and intelligence agencies as a new way of listening in, sometimes within the law, sometimes not. What’s different is the shift from targeting individuals to mass collection. In the 1990s, the FBI and others started vacuuming up data from phone calls and e-mails.  Snowden’s leaks showed how the combination of powerful data analysis tools, a loosened legal framework in the wake of the 9/11 attacks and the widespread adoption of smartphones laid the groundwork for data collection on a previously unimaginable scale. The NSA and the FBI still need a warrant to listen to the content of calls, but the metadata they collect about the calls can let them know who you talked to or e-mailed, when, from where and for how long, along with who those people call, and who their contacts call.

The Argument

In December, two federal judges looked at the NSA’s bulk collection of data. One called the technology “almost Orwellian” and said the program probably violated the Constitution. The other called it legal under existing precedents. Critics say what Snowden reveals is the twisting of that goal into the institutionalization of what author Gore Vidal called the “National Security State.” The officials who oversee the spy programs say they need to compile a giant haystack of data to find needles quickly; they dismiss privacy concerns as overblown. Director of National Intelligence James Clapper told Congress that snooping should be measured by a “peace-of-mind metric.” Supporters of the bulk collection say it’s justified by the 1979 Supreme Court decision, which held that since everyone knew the phone company kept records of their calls there was no expectation of privacy. Critics challenge that interpretation, and say that safeguards that may have been adequate in the days of dial telephones are meaningless in the age of smartphones. Revelations of the agency’s use of Heartbleed and other web bugs raised questions about how much insecurity the NSA was willing to create to keep Americans secure.

The Reference Shelf

  • Barack Obama’s August 2007 speech at Washington’s Wilson Center pledging that as president there would be “no more illegal wiretapping of American citizens.”
  • The bill limiting bulk collection of data by the NSA passed by the House of Representatives in May.
  • A December 2010 report from the American Civil Liberties Union on the impact of the Patriot Act on privacy rights.
  • Wired magazine’s 2012 article on the NSA’s metadata collection center in Utah.
  • The Louvre’s webpage for Rembrandt’s  17th-century painting, “Bathsheba at her Bath.”


First Published Jan. 23, 2014

To contact the writer of this QuickTake:

Roger Runningen in Washington at