Group items matching

in title, tags, annotations or url

The other shoe just dropped when it comes to how the federal government illegally spies on Americans. Last summer, the details of the NSA's "backdoor searches" were revealed. This involved big collections of content and metadata (so, no, not "just metadata" as meaningless as that phrase is) that were collected under Section 702 of the FISA Amendments Act (FAA). This is part of the program that the infamous PRISM effort operates under, and which allows the NSA to collect all sorts of content, including communications to, from or about a "target" -- where a "target" can be incredibly loosely defined (i.e., it can include groups or machines or just about anything). The "backdoor searches" were a special loophole added in 2011 allowing the NSA to make use of "US person names and identifiers as query terms." In the past, it had been limited (as per the NSA's mandate) to only non-US persons.

This morning, James Clapper finally responded to a request from Senator Ron Wyden concerning the number of such backdoor searches using US identifiers that were done by various government agencies. And, surprisingly, it's redaction free. The big reveal is... that it's not just the NSA doing these searches, but the CIA and FBI as well. This is especially concerning with regards to the FBI. This means that the FBI, who does surveillance on Americans, is spying on Americans communications that were collected by the NSA and that they're doing so without anything resembling a warrant. Oh, and let's make this even worse: the FBI isn't even tracking how often it does this. It's just doing it willy nilly:

Got that? Basically, the FBI often asks the NSA for a big chunk of data that the NSA probably shouldn't have in the first place -- including tons of Americans' communications, and the FBI gets to dump it into the same database that it is free to query. And the FBI tracks none of this, other than to say that it believes that there are a "substantial" number of such queries. This would seem to be a pretty blatant attempt to end run around the 4th Amendment, giving the FBI broad access to searching through the communications of Americans with what appears to be almost no oversight. Yikes! Oh, and it's not just the NSA, but the CIA as well. Remember, the CIA is not supposed to be doing any surveillance on US persons (like the NSA), but that's not what's happening at all. At least the CIA tracks some (but not all) of its abuse of backdoor searches:

Director of National Intelligence James Clapper has won the infamous Rosemary Award for worst open government performance in 2013, according to the citation published today by the National Security Archive at www.nsarchive.org. Despite heavy competition, Clapper's "No, sir" lie to Senator Ron Wyden's question: "Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?" sealed his receipt of the dubious achievement award, which cites the vastly excessive secrecy of the entire U.S. surveillance establishment. The Rosemary Award citation leads with what Clapper later called the "least untruthful" answer possible to congressional questions about the secret bulk collection of Americans' phone call data. It further cites other Clapper claims later proved false, such as his 2012 statement that "we don't hold data on U.S. citizens." But the Award also recognizes Clapper's fellow secrecy fetishists and enablers, including:

Gen. Keith Alexander, director of the NSA, for multiple Rose Mary Woods-type stretches, such as (1) claiming that the secret bulk collection prevented 54 terrorist plots against the U.S. when the actual number, according to the congressionally-established Privacy and Civil Liberties Oversight Board (PCLOB) investigation (pp. 145-153), is zero; (2) his 2009 declaration to the wiretap court that multiple NSA violations of the court's orders arose from differences over "terminology," an explanation which the chief judge said "strains credulity;" and (3) public statements by the NSA about its programs that had to be taken down from its website for inaccuracies (see Documents 78, 85, 87 in The Snowden Affair), along with public statements by other top NSA officials now known to be untrue (see "Remarks of Rajesh De," NSA General Counsel, Document 53 in The Snowden Affair).

Robert Mueller, former FBI director, for suggesting (as have Gen. Alexander and many others) that the secret bulk collection program might have been able to prevent the 9/11 attacks, when the 9/11 Commission found explicitly the problem was not lack of data points, but failing to connect the many dots the intelligence community already had about the would-be hijackers living in San Diego. The National Security Division lawyers at the Justice Department, for misleading their own Solicitor General (Donald Verrilli) who then misled (inadvertently) the U.S. Supreme Court over whether Justice let defendants know that bulk collection had contributed to their prosecutions. The same National Security Division lawyers who swore under oath in the Electronic Frontier Foundation's Freedom of Information Act lawsuit for a key wiretap court opinion that the entire text of the opinion was appropriately classified Top Secret/Sensitive Compartmented Information (release of which would cause "exceptionally grave damage" to U.S. national security). Only after the Edward Snowden leaks and the embarrassed governmental declassification of the opinion did we find that one key part of the opinion's text simply reproduced the actual language of the 4th Amendment to the U.S. Constitution, and the only "grave damage" was to the government's false claims.

In June 2013, the Guardian newspaper, utilizing documents disclosed by Edward Snowden, a former employee of a National Security Agency (NSA) contractor, reported that the FBI had obtained a ninety-day order from the Foreign Intelligence Surveillance Court (FISC) requiring Verizon Business to provide the NSA daily so-called telephone metadata on all their customers’ communications, although none were suspected of a connection with international terrorism or other wrongdoing. Later public revelations established that the order had been renewed thirty-six times since May 2006, and that companion FISC orders had been directed to all major telecommunications companies. This unprecedented intrusion into the activities that citizens heretofore considered private and personal is effected without any suspicion and without any limitation to information related to some known threat from a foreign actor considered dangerous to the United States. While the FISC has uniformly upheld the constitutionality of the dragnet telephony metadata and search program of the NSA in non-adversary proceedings, Article III courts are divided at present. The United States Supreme Court has recently declared that the Fourth Amendment should be interpreted today to secure the same level of privacy protection as was reasonably expected of citizens when the Amendment was ratified in 1792. In making that assessment, law enforcement resources, investigative priorities, and technological and jurisdictional limitations on the government are all pertinent. As elaborated in the analysis linked below, the historical interpretation of the Fourth Amendment’s privacy guarantees suggests that the NSA’s bulk collection of telephone metadata violates the Constitution.   Click here to view The Rutherford Institute's historic analysis of the Fourth Amendment as it relates to the NSA's surveillance activities.

James Comey famously threatened to resign from the Justice Department in 2004 over the warrantless surveillance of Americans' internet records. But once Justice Department and National Security Agency lawyers found a novel legal theory to cover the surveillance, the man Barack Obama tapped last week to lead the FBI stayed on as deputy attorney general for another year as the monitoring continued.Comey was the acting attorney general in March 2004, when long-simmering legal tensions over the online "metadata" surveillance pitted the Justice Department and FBI against the Bush White House and NSA. That incident, dramatically recounted by Comey to the Senate in May 2007, earned the 6ft 8in former federal prosecutor a reputation for integrity that has become central to his persona.

President Obama directly referred to that reputation when he nominated Comey to take over the FBI on June 21. Hovering over the announcement were the Guardian and Washington Post's revelations of wide-ranging surveillance efforts."To know Jim Comey is also to know his fierce independence and his deep integrity," Obama said. "He was prepared to give up a job he loved rather than be part of something he felt was fundamentally wrong."Except that a classified report recounting the incident, acquired by the Guardian, complicates that view. Comey threatened to resign over the perceived illegality of one aspect of the surveillance. But he remained at the Justice Department for another year as that effort, operating under a new legal theory, continued nearly unchanged.

Comey would later testify to the Senate that the episode was "the most difficult of my professional career."But "immediately," the NSA IG report shows, lawyers from the NSA and Comey's Justice Department "began efforts to recreate this authority." They found it in what the document nebulously refers to as a Pen Register/Trap and Trace Order – a reference to devices traditionally used by surveillance officials to record the incoming and outgoing calls made and received by a telephone.The Fisa court, the secret court that oversees NSA surveillance, approved the first such order for NSA to again collect and analyze large volumes of internet records from Americans on July 14 2004, barely three months after Comey's rebellion.

Britain's spy agency GCHQ has secretly gained access to the network of cables which carry the world's phone calls and internet traffic and has started to process vast streams of sensitive personal information which it is sharing with its American partner, the National Security Agency (NSA).The sheer scale of the agency's ambition is reflected in the titles of its two principal components: Mastering the Internet and Global Telecoms Exploitation, aimed at scooping up as much online and telephone traffic as possible. This is all being carried out without any form of public acknowledgement or debate.One key innovation has been GCHQ's ability to tap into and store huge volumes of data drawn from fibre-optic cables for up to 30 days so that it can be sifted and analysed. That operation, codenamed Tempora, has been running for some 18 months.

GCHQ and the NSA are consequently able to access and process vast quantities of communications between entirely innocent people, as well as targeted suspects.This includes recordings of phone calls, the content of email messages, entries on Facebook and the history of any internet user's access to websites – all of which is deemed legal, even though the warrant system was supposed to limit interception to a specified range of targets.The existence of the programme has been disclosed in documents shown to the Guardian by the NSA whistleblower Edward Snowden as part of his attempt to expose what he has called "the largest programme of suspicionless surveillance in human history"."It's not just a US problem. The UK has a huge dog in this fight," Snowden told the Guardian. "They [GCHQ] are worse than the US."

However, on Friday a source with knowledge of intelligence argued that the data was collected legally under a system of safeguards, and had provided material that had led to significant breakthroughs in detecting and preventing serious crime.Britain's technical capacity to tap into the cables that carry the world's communications – referred to in the documents as special source exploitation – has made GCHQ an intelligence superpower.By 2010, two years after the project was first trialled, it was able to boast it had the "biggest internet access" of any member of the Five Eyes electronic eavesdropping alliance, comprising the US, UK, Canada, Australia and New Zealand.UK officials could also claim GCHQ "produces larger amounts of metadata than NSA". (metadata describes basic information on who has been contacting whom, without detailing the content.)By May last year 300 analysts from GCHQ, and 250 from the NSA, had been assigned to sift through the flood of data.The Americans were given guidelines for its use, but were told in legal briefings by GCHQ lawyers: "We have a light oversight regime compared with the US".

"In a review of NSA surveillance last month, President Obama called for a new approach on telephony metadata that will 'establish a mechanism that preserves the capabilities we need without the government holding this bulk metadata.' Obama said that a third party holding all the data in a single, consolidated database would be essentially doing what is a government function, and may not increase public confidence that its privacy is being protected. Now, an RFI (request for information) has been posted to get information on U.S. industry's commercially available capabilities, so that the government can investigate alternative approaches."

A special unit operating under cover and protected by diplomatic immunity, assigned to a very sensitive mission: to spy on the communication of the Italian leadership. That is what top secret documents leaked by Edward Snowden and published in Italy exclusively by l'Espresso in collaboration with "la Repubblica" reveal. A file mentions the "Special Collection Service " (SCS) sites in Rome and in Milan, the very same service which, according to the German weekly "Der Spiegel ", spied on the mobile phone of the German Chancellor, Angela Merkel. "Special Collection Sites", reads the file published today by l'Espresso, "provide considerable perishable intelligence on leadership communications largely facilitated by site presence within a national capital". These documents are very important because they contradict recent statements by the Italian Prime Minister reassuring the Italian Parliament. Speaking to the Chamber of Deputies four weeks ago, Enrico Letta said: "Based on the analysis conducted by our intelligence services and our international contacts, we are not aware that the security of the communications of the Italian government and embassies has been compromised, nor are we aware that the privacy of Italian citizens has been compromised". These top secret documents tell a different story, however.

The Special Collection Service is likely one of the most sensitive units in U.S. intelligence. The service deploys teams under diplomatic cover, operating in US embassies around the world to control friendly and enemy governments. The top secret NSA document examined by l'Espresso reveals that "in 1988 [SCS] had 88 sites, our peak". The SCS is assigned to a special mission: monitoring the communications of the political, and likely economical, leaders of host nations. For this reason, SCS teams operate within the heart of power: in embassies and consulates, working in close collaboration with the CIA. Also in Rome, in the US embassy located in via Veneto, from those very same roofs which witnessed the Dolce Vita. Snowden's files reveal that, at least until 2010, the Special Collection Service maintained two sites in Italy: one in Rome, a base staffed with agents, and one in Milan, the capital of the Italian economy where, according to a file dated 2010 and originally published in Der Spiegel, the SCS would run an unmanned site. Two sites in a relatively small country like Italy is unusual: only in Germany -- a prime target for NSA in Europe -- does SCS maintain two bases.

NSA's mass spying activities did not target our leadership and diplomacy alone, but it possibly also targeted millions of Italian citizens. A file on the top secret programme "Boundless Informant" that is labeled "Italy" reveals that between December 10, 2012 and January 9, 2013, the NSA collected the metadata for 45.893.570 telephone calls. Estimates close to this figure had already circulated, but now the actual document indicates the penetration of this monitoring. All of the metadata gathered in our country between December 10^th and January 9^th 2013 as reflected on this slide relate to phone communication, unlike the slides published in Germany and France, where internet communication metadata were targeted as well.

A member of the White House review panel on NSA surveillance said he was “absolutely” surprised when he discovered the agency’s lack of evidence that the bulk collection of telephone call records had thwarted any terrorist attacks.

“It was, ‘Huh, hello? What are we doing here?’” said Geoffrey Stone, a University of Chicago law professor, in an interview with NBC News. “The results were very thin.”While Stone said the mass collection of telephone call records was a “logical program” from the NSA’s perspective, one question the White House panel was seeking to answer was whether it had actually stopped “any [terror attacks] that might have been really big.” Advertise | AdChoices “We found none,” said Stone. Under the NSA program, first revealed by ex-contractor Edward Snowden, the agency collects in bulk the records of the time and duration of phone calls made by persons inside the United States.Stone was one of five members of the White House review panel – and the only one without any intelligence community experience – that this week produced a sweeping report recommending that the NSA’s collection of phone call records be terminated to protect Americans’ privacy rights.The panel made that recommendation after concluding that the program was “not essential in preventing attacks.”“That was stunning. That was the ballgame,” said one congressional intelligence official, who asked not to be publicly identified. “It flies in the face of everything that they have tossed at us.”

The conclusions of the panel’s reports were at direct odds with public statements by President Barack Obama and U.S. intelligence officials. “Lives have been saved,” Obama told reporters last June, referring to the bulk collection program and another program that intercepts communications overseas. “We know of at least 50 threats that have been averted because of this information.”

The Drug Enforcement Administration is collecting information about more than just license plates with the tracking system revealed by the American Civil Liberties Union. Documents released by the ACLU this morning show that the DEA is also using the license plate readers (LPRs) on which this system relies to capture photographs of a vehicles’ passengers. The images can then be run through facial recognition software. This is meant to give the DEA more context about the people whose movements it’s tracking with this program, which gathers data from more than 100 LPRs managed by an unknown number of police departments around the country to aid in their investigations. The program was originally meant to assist with civil asset forfeiture cases, but it has since expanded to assist departments approved by the El Paso Intelligence Center with investigations into murders, rapes, and other crimes, the Wall Street Journal reports.

Previous reports indicated that the DEA was collecting license plate information about “millions” of Americans. That figure might be low if it didn’t account for the number of plates collected versus the number of people in a vehicle when these images are taken. Either way, this program represents a clear violation of privacy for many Americans, most of whom didn’t know the DEA could collect this information. As I wrote before: The result is a national surveillance program with an unknown number of contributors offering up location data about millions of Americans; all to a database used by an untold number of police departments without any public oversight regarding their searches.

That’s a problem. Backchannel reported in December that police have used their access to license plate readers to stalk former colleagues, and IB Times revealed earlier this month that Gov. Chris Christie (R-NJ) used location data to smear a political rival. Perhaps the DEA will support the program by claiming that learning who is in a vehicle isn’t much different from learning where the vehicle was going — it could all be considered metadata, and the government considers that information to be fair game.

The first (and thus far only) roll-back of post-9/11 surveillance authorities was implemented over the weekend: The National Security Agency shuttered its program for collecting and holding the metadata of Americans’ phone calls under Section 215 of the Patriot Act. While bulk collection under Section 215 has ended, the government can obtain access to this information under the procedures specified in the USA Freedom Act. Indeed, some experts have argued that the Agency likely has access to more metadata because its earlier dragnet didn’t cover cell phones or Internet calling. In addition, the metadata of calls made by an individual in the United States to someone overseas and vice versa can still be collected in bulk — this takes place abroad under Executive Order 12333. No doubt the NSA wishes that this was the end of the surveillance reform story and the Paris attacks initially gave them an opening. John Brennan, the Director of the CIA, implied that the attacks were somehow related to “hand wringing” about spying and Sen. Tom Cotton (R-Ark.) introduced a bill to delay the shut down of the 215 program. Opponents of encryption were quick to say: “I told you so.”

But the facts that have emerged thus far tell a different story. It appears that much of the planning took place IRL (that’s “in real life” for those of you who don’t have teenagers). The attackers, several of whom were on law enforcement’s radar, communicated openly over the Internet. If France ever has a 9/11 Commission-type inquiry, it could well conclude that the Paris attacks were a failure of the intelligence agencies rather than a failure of intelligence authorities. Despite the passage of the USA Freedom Act, US surveillance authorities have remained largely intact. Section 702 of the FISA Amendments Act — which is the basis of programs like PRISM and the NSA’s Upstream collection of information from Internet cables — sunsets in the summer of 2017. While it’s difficult to predict the political environment that far out, meaningful reform of Section 702 faces significant obstacles. Unlike the Section 215 program, which was clearly aimed at Americans, Section 702 is supposedly targeted at foreigners and only picks up information about Americans “incidentally.” The NSA has refused to provide an estimate of how many Americans’ information it collects under Section 702, despite repeated requests from lawmakers and most recently a large cohort of advocates. The Section 215 program was held illegal by two federal courts (here and here), but civil attempts to challenge Section 702 have run into standing barriers. Finally, while two review panels concluded that the Section 215 program provided little counterterrorism benefit (here and here), they found that the Section 702 program had been useful.

There is, nonetheless, some pressure to narrow the reach of Section 702. The recent decision by the European Court of Justice in the safe harbor case suggests that data flows between Europe and the US may be restricted unless the PRISM program is modified to protect the information of Europeans (see here, here, and here for discussion of the decision and reform options). Pressure from Internet companies whose business is suffering — estimates run to the tune of $35 to 180 billion — as a result of disclosures about NSA spying may also nudge lawmakers towards reform. One of the courts currently considering criminal cases which rely on evidence derived from Section 702 surveillance may hold the program unconstitutional either on the basis of the Fourth Amendment or Article III for the reasons set out in this Brennan Center report. A federal district court in Colorado recently rejected such a challenge, although as explained in Steve’s post, the decision did not seriously explore the issues. Further litigation in the European courts too could have an impact on the debate.

Were Israeli companies Verint and Narus the ones that collected information from the U.S. communications network for the National Security Agency? The question arises amid controversy over revelations that the NSA has been collecting the phone records of hundreds of millions of Americans every day, creating a database through which it can learn whether terror suspects have been in contact with people in the United States. It also was disclosed this week that the NSA has been gathering all Internet usage - audio, video, photographs, emails and searches - from nine major U.S. Internet providers, including Microsoft and Google, in hopes of detecting suspicious behavior that begins overseas.

According to an article in the American technology magazine "Wired" from April 2012, two Israeli companies – which the magazine describes as having close connections to the Israeli security community – conduct bugging and wiretapping for the NSA. Verint, which took over its parent company Comverse Technology earlier this year, is responsible for tapping the communication lines of the American telephone giant Verizon, according to a past Verizon employee sited by James Bamford in Wired. Neither Verint nor Verizon commented on the matter.

Natus, which was acquired in 2010 by the American company Boeing, supplied the software and hardware used at AT&T wiretapping rooms, according to whistleblower Mark Klein, who revealed the information in 2004. Klein, a past technician at AT&T who filed a suit against the company for spying on its customers, revealed a "secret room" in the company's San Fransisco office, where the NSA collected data on American citizens' telephone calls and Internet surfing. Klein's claims were reinforced by former NSA employee Thomas Drake who testified that the agency uses a program produced by Narus to save the personal electrical communications of AT&T customers.  Both Verint and Narus have ties to the Israeli intelligence agency and the Israel Defense Forces intelligence-gathering unit 8200. Hanan Gefen, a former commander of the 8200 unit, told Forbes magazine in 2007 that Comverse's technology, which was formerly the parent company of Verint and merged with it this year, was directly influenced by the technology of 8200. Ori Cohen, one of the founders of Narus, told Fortune magazine in 2001 that his partners had done technology work for the Israeli intelligence.

Senior figures inside British intelligence have been alarmed by GCHQ's secret decision to tap into transatlantic cables in order to engage in the bulk interception of phone calls and internet traffic.According to one source who has been directly involved in GCHQ operations, concerns were expressed when the project was being discussed internally in 2008: "We felt we were starting to overstep the mark with some of it. People from MI5 were complaining that they were going too far from a civil liberties perspective … We all had reservations about it, because we all thought: 'If this was used against us, we wouldn't stand a chance'."The Guardian revealed on Friday that GCHQ has placed more than 200 probes on transatlantic cables and is processing 600m "telephone events" a day as well as up to 39m gigabytes of internet traffic. Using a programme codenamed Tempora, it can store and analyse voice recordings, the content of emails, entries on Facebook, the use of websites as well as the "metadata" which records who has contacted who. The programme is shared with GCHQ's American partner, the National Security Agency.

Interviews with the UK source and the NSA whistleblower Edward Snowden raise questions about whether the programme:■ Exploits existing law which was passed by parliament without any anticipation that it would be used for this purpose.■ For the first time allows GCHQ to process bulk internal UK traffic which is routed overseas via these cables.■ Allows the NSA to engage in bulk intercepts of internal US traffic which would be forbidden in its own territory.■ Functions with no effective oversight.

The source claimed that even the conventional warrant system has been distorted – whereas police used to ask for a warrant before intercepting a target's communications, they will now ask GCHQ to intercept the target's communications and then use that information to seek a warrant.There is a particular concern that the programme allows GCHQ to break the boundary which stopped it engaging in the bulk interception of internal UK communications. The Ripa requirement that one end of a communication must be outside the UK was a significant restriction when it was applied to phone calls using satellites, but it is no longer effective in the world of fibre-optic cables. "The point is that this is an island," the source said. "Everything comes and goes – nearly everything – down fibre-optic cables. You make a mobile phone call, it goes to a mast and then down into a fibre-optic cable, under the ground and away. And even if the call is UK to UK, it's very likely – because of the way the system is structured – to go out of the UK and come back in through these fibre-optic channels."

Twenty-two organizations including Unitarian church groups, gun ownership advocates, and a broad coalition of membership and political advocacy organizations filed suit against the National Security Agency for violating their First Amendment right of association by illegally collecting their call records. The coalition is represented by EFF. At the heart of First Unitarian Church of Los Angeles v. NSA is the bulk telephone records collection program that was confirmed by the publication of an order from the Foreign Intelligence Surveillance Court (FISC) in June of 2013. The Director of National Intelligence (DNI) further confirmed that this formerly secret document was authentic, and part of a broader program to collect all major telecommunications customers’ call history. The order demands wholesale collection of every call made, the location of the phone, the time of the call, the duration of the call, and other “identifying information” for every phone and call for all customers of Verizon for a period of three months. Government officials further confirmed that this was just one of series of orders issued on a rolling basis since at least 2006. First Unitarian v. NSA argues that this spying violates the First Amendment, which protects the freedom to associate and express political views as a group.

Twenty-two organizations including Unitarian church groups, gun ownership advocates, and a broad coalition of membership and political advocacy organizations filed suit against the National Security Agency for violating their First Amendment right of association by illegally collecting their call records. The coalition is represented by EFF. At the heart of First Unitarian Church of Los Angeles v. NSA is the bulk telephone records collection program that was confirmed by the publication of an order from the Foreign Intelligence Surveillance Court (FISC) in June of 2013. The Director of National Intelligence (DNI) further confirmed that this formerly secret document was authentic, and part of a broader program to collect all major telecommunications customers’ call history. The order demands wholesale collection of every call made, the location of the phone, the time of the call, the duration of the call, and other “identifying information” for every phone and call for all customers of Verizon for a period of three months. Government officials further confirmed that this was just one of series of orders issued on a rolling basis since at least 2006. First Unitarian v. NSA argues that this spying violates the First Amendment, which protects the freedom to associate and express political views as a group.

The case challenges the mass telephone records collection that was confirmed by the FISA Order that was published on June 5, 2013 and confirmed by the Director of National Intelligence (DNI) on June 6, 2013. The DNI confirmed that the collection was “broad in scope” and conducted under the “business records” provision of the Foreign Intelligence Surveillance Act, also known as section 215 of the Patriot Act and 50 U.S.C. section 1861. The facts have long been part of EFF’s Jewel v. NSA case. The case does not include section 702 programs, which includes the recently made public and called the PRISM program or the fiber optic splitter program that is included (along with the telephone records program) in the Jewel v. NSA case. 

The National Security Agency has collected almost 200 million text messages a day from across the globe, using them to extract data including location, contact networks and credit card details, according to top-secret documents. The untargeted collection and storage of SMS messages – including their contacts – is revealed in a joint investigation between the Guardian and the UK’s Channel 4 News based on material provided by NSA whistleblower Edward Snowden. The documents also reveal the UK spy agency GCHQ has made use of the NSA database to search the metadata of “untargeted and unwarranted” communications belonging to people in the UK.

The NSA program, codenamed Dishfire, collects “pretty much everything it can”, according to GCHQ documents, rather than merely storing the communications of existing surveillance targets. The NSA has made extensive use of its vast text message database to extract information on people’s travel plans, contact books, financial transactions and more – including of individuals under no suspicion of illegal activity. An agency presentation from 2011 – subtitled “SMS Text Messages: A Goldmine to Exploit” – reveals the program collected an average of 194 million text messages a day in April of that year. In addition to storing the messages themselves, a further program known as “Prefer” conducted automated analysis on the untargeted communications.

The Prefer program uses automated text messages such as missed call alerts or texts sent with international roaming charges to extract information, which the agency describes as “content-derived metadata”, and explains that “such gems are not in current metadata stores and would enhance current analytics”. On average, each day the NSA was able to extract:

Telephone and Internet metadata are protected by law under the aforementioned “pen register” statute.  That statute says that “no person may install or use a pen register or a trap and trace device without first obtaining a court order under section 3123 of this title or under the Foreign Intelligence Surveillance Act of 1978 (50 U.S.C. 1801 et seq.).” Pen registers and trap and trace devices collect dialing, routing, signaling or addressing information. Violation of the statute is a criminal misdemeanor.

(AP) — The U.S. government is looking at ways to prevent anyone from spying on its own surveillance of Americans' phone records. As the Obama administration considers shifting the collection of those records from the National Security Agency to requiring that they be stored at phone companies or elsewhere, it's quietly funding research to prevent phone company employees or eavesdroppers from seeing whom the U.S. is spying on, The Associated Press has learned. The Office of the Director of National Intelligence has paid at least five research teams across the country to develop a system for high-volume, encrypted searches of electronic records kept outside the government's possession. The project is among several ideas that would allow the government to discontinue storing Americans' phone records, but still search them as needed.

Under the research, U.S. data mining would be shielded by secret coding that could conceal identifying details from outsiders and even the owners of the targeted databases, according to public documents obtained by The Associated Press and AP interviews with researchers, corporate executives and government officials.

Internal documents describing the Security and Privacy Assurance Research project do not cite the NSA or its phone surveillance program. But if the project were to prove successful, its encrypted search technology could pave the way for the government to shift storage of the records from NSA computers to either phone companies or a third-party organization. A DNI spokesman, Michael Birmingham, confirmed that the research was relevant to the NSA's phone records program. He cited "interest throughout the intelligence community" but cautioned that it may be some time before the technology is used. The intelligence director's office is by law exempt from disclosing detailed budget figures, so it's unclear how much money the government has spent on the SPAR project, which is overseen by the DNI's Intelligence Advanced Research Projects Activity office. Birmingham said the research is aimed for use in a "situation where a large sensitive data set is held by one party which another seeks to query, preserving privacy and enforcing access policies."

Citing the need to preserve evidence related to pending lawsuits, the Obama administration is asking for permission to keep data on billions of U.S. phone calls indefinitely instead of destroying it after five years. In a motion filed Tuesday with the Foreign Intelligence Surveillance Court, the Justice Department says the series of lawsuits over the program — including one filed by Sen. Rand Paul (R-Ky.) — create a duty for the government to hang on to the so-called metadata currently in the National Security Agency’s computer systems. “Based upon the issues raised by Plaintiffs in the … lawsuits and the Government’s potential defenses to those claims, the United States must ensure that all potentially relevant evidence is retained which includes the [business record] metadata obtained in bulk from certain telecommunications service providers pursuant to this Court’s production orders,” Justice Department lawyers write in a motion (posted here).

The motion was released Wednesday on the court’s public web page. There was no immediate indication of a ruling from the surveillance court. The NSA’s call metadata program is aimed at detecting terrorist plots affecting the U.S., but evidence of the effort’s success is murky. President Barack Obama has proposed ending the NSA’s collection of the data. Officials are considering storing it with the telephone companies themselves, creating a new entity to hold it, or reconstructing the program in some other way. The Wall Street Journal reported last week that U.S. officials were considering making the request submitted on Tuesday.

NSA phone data collection deemed legal: full ruling

A legal battle over the scope of US government surveillance took a turn in favour of the National Security Agency on Friday with a court opinion declaring that bulk collection of telephone data does not violate the constitution. The judgement, in a case brought before a district court in New York by the American Civil Liberties Union, directly contradicts the result of a similar challenge in a Washington court last week which ruled the NSA's bulk collection program was likely to prove unconstitutional and was "almost Orwellian" in scale. Friday's ruling makes it more likely that the issue will be settled by the US supreme court, although it may be overtaken by the decision of Barack Obama on whether to accept the recommendations of a White House review panel to ban the NSA from directly collecting such data. But the ruling from Judge William Pauley, a Clinton appointee to the Southern District of New York, will provide important ammunition for those within the intelligence community urging Obama to maintain the programme.

Judge Pauley said privacy protections enshrined in the fourth amendment of the US constitution needed to be balanced against a government need to maintain a database of records to prevent future terrorist attacks. “The right to be free from searches is fundamental but not absolute,” he said. “Whether the fourth amendment protects bulk telephony metadata is ultimately a question of reasonableness.”

President Barack Obama has conceded that mass collection of private data by the US government may be unnecessary and said there were different ways of “skinning the cat”, which could allow intelligence agencies to keep the country safe without compromising privacy. In an apparent endorsement of a recommendation by a review panel to shift responsibility for the bulk collection of telephone records away from the National Security Agency and on to the phone companies, the president said change was necessary to restore public confidence. “In light of the disclosures, it is clear that whatever benefits the configuration of this particular programme may have, may be outweighed by the concerns that people have on its potential abuse,” Obama told an end-of-year White House press conference. “If it that’s the case, there may be a better way of skinning the cat.”

Though insisting he will not make a final decision until January, this is the furthest the president has gone in backing calls to dismantle the programme to collect telephone data, a practice the NSA claims has legal foundation under section 215 of the Patriot Act. This week, a federal judge said the program “very likely” violates the US constitution. “There are ways we can do this potentially that give people greater assurance that there are checks and balances, sufficient oversight and transparency,” Obama added. “Programmes like 215 could be redesigned in ways that give you the same information when you need it without creating these potentials for abuse. That’s exactly what we should be doing: to evaluate things in a very clear specific way and moving forward on changes. And that’s what I intend to do.”

The president would not comment on a suggestion last weekend by Richard Ledgett, the NSA official investigating the Snowden leaks, that an amnesty might be appropriate in exchange for the return of the data Snowden took from the agency. Obama said he could not comment specifically because Snowden was “under indictment”, something not previously disclosed. While the Justice Department filed a criminal complaint against Snowden on espionage-related charges in June, there has been no public subsequent indictment, although it is possible one exists under gag order. The Justice Department referred comment on a Snowden indictment to the White House. Caitlin Hayden, the chief spokeswoman for the White House National Security Council, clarified that Obama was referring to the criminal complaint against Snowden. It remains unclear if there is an indictment under seal.