Group items tagged

A review of top-secret NSA documents suggests that the surveillance agency still collects and sifts through large quantities of Americans' online data – despite the Obama administration's insistence that the program that began under Bush ended in 2011.Shawn Turner, the Obama administration's director of communications for National Intelligence, told the Guardian that "the internet metadata collection program authorized by the Fisa court was discontinued in 2011 for operational and resource reasons and has not been restarted."But the documents indicate that the amount of internet metadata harvested, viewed, processed and overseen by the Special Source Operations (SSO) directorate inside the NSA is extensive.While there is no reference to any specific program currently collecting purely domestic internet metadata in bulk, it is clear that the agency collects and analyzes significant amounts of data from US communications systems in the course of monitoring foreign targets.

On December 26 2012, SSO announced what it described as a new capability to allow it to collect far more internet traffic and data than ever before. With this new system, the NSA is able to direct more than half of the internet traffic it intercepts from its collection points into its own repositories. One end of the communications collected are inside the United States.The NSA called it the "One-End Foreign (1EF) solution". It intended the program, codenamed EvilOlive, for "broadening the scope" of what it is able to collect. It relied, legally, on "FAA Authority", a reference to the 2008 Fisa Amendments Act that relaxed surveillance restrictions.This new system, SSO stated in December, enables vastly increased collection by the NSA of internet traffic. "The 1EF solution is allowing more than 75% of the traffic to pass through the filter," the SSO December document reads. "This milestone not only opened the aperture of the access but allowed the possibility for more traffic to be identified, selected and forwarded to NSA repositories."

It continued: "After the EvilOlive deployment, traffic has literally doubled."The scale of the NSA's metadata collection is highlighted by references in the documents to another NSA program, codenamed ShellTrumpet.On December 31, 2012, an SSO official wrote that ShellTrumpet had just "processed its One Trillionth metadata record".

HERE WAS A SIMPLE AIM at the heart of the top-secret program: Record the website browsing habits of “every visible user on the Internet.” Before long, billions of digital records about ordinary people’s online activities were being stored every day. Among them were details cataloging visits to porn, social media and news websites, search engines, chat forums, and blogs. The mass surveillance operation — code-named KARMA POLICE — was launched by British spies about seven years ago without any public debate or scrutiny. It was just one part of a giant global Internet spying apparatus built by the United Kingdom’s electronic eavesdropping agency, Government Communications Headquarters, or GCHQ. The revelations about the scope of the British agency’s surveillance are contained in documents obtained by The Intercept from National Security Agency whistleblower Edward Snowden. Previous reports based on the leaked files have exposed how GCHQ taps into Internet cables to monitor communications on a vast scale, but many details about what happens to the data after it has been vacuumed up have remained unclear.

Amid a renewed push from the U.K. government for more surveillance powers, more than two dozen documents being disclosed today by The Intercept reveal for the first time several major strands of GCHQ’s existing electronic eavesdropping capabilities.

The surveillance is underpinned by an opaque legal regime that has authorized GCHQ to sift through huge archives of metadata about the private phone calls, emails and Internet browsing logs of Brits, Americans, and any other citizens — all without a court order or judicial warrant

Canada’s electronic surveillance agency is covertly monitoring vast amounts of Canadians’ emails as part of a sweeping domestic cybersecurity operation, according to top-secret documents. The surveillance initiative, revealed Wednesday by CBC News in collaboration with The Intercept, is sifting through millions of emails sent to Canadian government agencies and departments, archiving details about them on a database for months or even years. The data mining operation is carried out by the Communications Security Establishment, or CSE, Canada’s equivalent of the National Security Agency. Its existence is disclosed in documents obtained by The Intercept from NSA whistleblower Edward Snowden. The emails are vacuumed up by the Canadian agency as part of its mandate to defend against hacking attacks and malware targeting government computers. It relies on a system codenamed PONY EXPRESS to analyze the messages in a bid to detect potential cyber threats.

Last year, CSE acknowledged it collected some private communications as part of cybersecurity efforts. But it refused to divulge the number of communications being stored or to explain for how long any intercepted messages would be retained. Now, the Snowden documents shine a light for the first time on the huge scope of the operation — exposing the controversial details the government withheld from the public. Under Canada’s criminal code, CSE is not allowed to eavesdrop on Canadians’ communications. But the agency can be granted special ministerial exemptions if its efforts are linked to protecting government infrastructure — a loophole that the Snowden documents show is being used to monitor the emails. The latest revelations will trigger concerns about how Canadians’ private correspondence with government employees are being archived by the spy agency and potentially shared with police or allied surveillance agencies overseas, such as the NSA. Members of the public routinely communicate with government employees when, for instance, filing tax returns, writing a letter to a member of parliament, applying for employment insurance benefits or submitting a passport application.

Chris Parsons, an internet security expert with the Toronto-based internet think tank Citizen Lab, told CBC News that “you should be able to communicate with your government without the fear that what you say … could come back to haunt you in unexpected ways.” Parsons said that there are legitimate cybersecurity purposes for the agency to keep tabs on communications with the government, but he added: “When we collect huge volumes, it’s not just used to track bad guys. It goes into data stores for years or months at a time and then it can be used at any point in the future.” In a top-secret CSE document on the security operation, dated from 2010, the agency says it “processes 400,000 emails per day” and admits that it is suffering from “information overload” because it is scooping up “too much data.” The document outlines how CSE built a system to handle a massive 400 terabytes of data from Internet networks each month — including Canadians’ emails — as part of the cyber operation. (A single terabyte of data can hold about a billion pages of text, or about 250,000 average-sized mp3 files.)

The latest revelations will add to the intense public and congressional debate around the extent of NSA surveillance programs. They come as senior intelligence officials testify to the Senate judiciary committee on Wednesday, releasing classified documents in response to the Guardian's earlier stories on bulk collection of phone records and Fisa surveillance court oversight.

The National Security Agency is secretly providing data to nearly two dozen U.S. government agencies with a “Google-like” search engine built to share more than 850 billion records about phone calls, emails, cellphone locations, and internet chats, according to classified documents obtained by The Intercept. The documents provide the first definitive evidence that the NSA has for years made massive amounts of surveillance data directly accessible to domestic law enforcement agencies. Planning documents for ICREACH, as the search engine is called, cite the Federal Bureau of Investigation and the Drug Enforcement Administration as key participants. ICREACH contains information on the private communications of foreigners and, it appears, millions of records on American citizens who have not been accused of any wrongdoing. Details about its existence are contained in the archive of materials provided to The Intercept by NSA whistleblower Edward Snowden. Earlier revelations sourced to the Snowden documents have exposed a multitude of NSA programs for collecting large volumes of communications. The NSA has acknowledged that it shares some of its collected data with domestic agencies like the FBI, but details about the method and scope of its sharing have remained shrouded in secrecy.

ICREACH has been accessible to more than 1,000 analysts at 23 U.S. government agencies that perform intelligence work, according to a 2010 memo. A planning document from 2007 lists the DEA, FBI, Central Intelligence Agency, and the Defense Intelligence Agency as core members. Information shared through ICREACH can be used to track people’s movements, map out their networks of associates, help predict future actions, and potentially reveal religious affiliations or political beliefs. The creation of ICREACH represented a landmark moment in the history of classified U.S. government surveillance, according to the NSA documents.

Documents published with this article: CIA Colleagues Enthusiastically Welcome NSA Training Sharing Communications Metadata Across the U.S. Intelligence Community CRISSCROSS/PROTON Point Paper Decision Memorandum for the DNI on ICREACH Metadata Sharing Memorandum Sharing SIGINT metadata on ICREACH Metadata Policy Conference ICREACH Wholesale Sharing Black Budget Extracts

The U.S. government has acknowledged that it swept up huge volumes of data from emails in the U.S. for several years without any court approval, based solely on the orders of former President George W. Bush. In a court filings on Monday, government lawyers said that the Internet program ran in parallel with a program gathering so-called metadata about telephone calls. The counterterrorism efforts operated under presidential authority before a judge approved them in July 2004, said a 2007 court filing made public Monday by the Justice Department (and posted here.)

"After the 9/11 attacks and pursuant to an authorization of the President, [redacted] the NSA [redacted] the bulk collection of non-content information about  telephone calls and Internet communications (hereafter 'metadata') activities that enable the NSA to uncover the contacts [redacted] of members or agents of al Qaeda or affiliated terrorist organizations," a senior NSA official wrote in an October 2007 declaration originally filed under seal as part of an effort to defeat litigation about the snooping Bush ordered. "Specifically, the President authorized the the NSA to collect metadata related to Internet communications for the purpose of conducting targeted analysis to track Al Qaeda-related networks. Internet metadata is header/router/addressing information, such as the 'to,' 'from,' 'cc,' and 'bcc' lines, as opposed to the body or 're' lines, of a standard e-mail. Since July 2004, the collection of Internet metadata has been conducted pursuant to an Order of the Foreign Intelligence Surveillance Court," the still-unidentified official from NSA's Signals Intelligence Directorate continued. The email program was effectively public since June of last year, after contractor Edward Snowden leaked a top-secret National Security Agency inspector general report that described the program.

FISC Judge Colleen Kollar-Kotelly's opinion approving the surveillance was officially released in November 2013. However, the date she issued it was redacted. Many surmised that her opinion followed a dust-up in March 2004, when then-Deputy Attorney General James Comey questioned the legality of some aspect of Bush's post-9/11 surveillance programs and refused to reauthorize that portion of the surveillance. Comey's refusal is said to have put the program into turmoil for a period of months, until officials sought and won the order from Kollar-Kotelly blessing the gathering of both the email and telephone metadata. The publicly released version of Kollar-Kotelly's opinion does not discuss the operation of the program during the period before the application for court approval. The filings Monday came in continuing legal wrangling over obligations pending lawsuits may create for the NSA to hang on to aging metadata that it would ordinarily have been required to erase under FISC orders. A federal judge in San Francisco has required that the NSA preserve that data, at least for now, rather than erasing it.

The NSA’s spying on everyone’s metadata can tell them just about everything about us … and it violates our Constitutional right to freedom of association. But people are getting distracted from the big picture by focusing on metadata. As security expert Bruce Schneier wrote yesterday: What frustrates me about all of this — [the Privacy and Civil Liberties Oversight Board] report, the president’s speech, and so many other things — is that they focus on the bulk collection of cell phone call records. There’s so much more bulk collection going on — phone calls, e-mails, address books, buddy lists, text messages, cell phone location data, financial documents, calendars, [smartphone apps] etc. — and we really need legislation and court opinions on it all. But because cell phone call records were the first disclosure, they’re what gets the attention. Indeed, Schneier confirmed last October what we’ve been saying for years … don’t get too distracted by the details, because the government is spying on everything:

Honestly, I think the details matter less and less. We have to assume that the NSA has EVERYONE who uses electronic communications under CONSTANT surveillance. New details about hows and whys will continue to emerge …but the big picture will remain the same. He’s right. As just one example, there is substantial evidence from top NSA and FBI whistleblowers that the government is recording the content of our calls and emails … word-for-word. So what should we make of the government’s denials that it records content? Given that the government has been caught lying about spying again and again, I’m not sure how much weight we should give to such denials. NSA whistleblower Russ Tice notes: They’re collecting content … word-for-word. *** You can’t trust these people. They lie, and they lie a lot.

This whole morning, while all these stories of the NSA hacking directly into Google and Yahoo's network have been popping up, I've been at the Inbox Love conference, all about the future of email. The "keynote" that just concluded, was Ladar Levison from Lavabit (with an assist from Mike Janke from Silent Circle), talking about the just announced Dark Mail Alliance, between Lavabit and Silent Circle -- the other "security" focused communications company who shut down its email offering after Lavabit was forced to shut down. Levison joked that they went with "Dark Mail" because "Black Mail" might have negative connotations. Perhaps just as interesting, Levison is going to be releasing the Lavabit source code (and doing a Kickstarter project to support this), with the hope that many others can set up their own secure email using Lavabit's code, combined with the new Dark Mail Alliance secure technology which will be available next year. As noted, the Alliance is working on trying to create truly secure and surveillance-proof email. Of course, nothing is ever 100% surveillance proof -- and both members of the alliance have previously claimed that it was almost impossible to do surveillance-proof email. However, they're claiming they've had a "breakthrough" that will help.

The newly developed technology has been designed to look just like ordinary email, with an interface that includes all the usual folders—inbox, sent mail, and drafts. But where it differs is that it will automatically deploy peer-to-peer encryption, so that users of the Dark Mail technology will be able to communicate securely. The encryption, based on a Silent Circle instant messaging protocol called SCIMP, will apply to both content and metadata of the message and attachments. And the secret keys generated to encrypt the communications will be ephemeral, meaning they are deleted after each exchange of messages. For the NSA and similar surveillance agencies across the world, it will sound like a nightmare. The technology will thwart attempts to sift emails directly from Internet cables as part of so-called “upstream” collection programs and limit the ability to collect messages directly from Internet companies through court orders. Covertly monitoring encrypted Dark Mail emails would likely have to be done by deploying Trojan spyware on a targeted user. If every email provider in the world adopted this technology for all their users, it would render dragnet interception of email messages and email metadata virtually impossible.

Importantly, they're not asking everyone to just trust them to be secure -- even though both companies have the right pedigree to deserve some level of trust. Instead, they're going to release the source code for public scrutiny and audits, and they're hoping that other email providers will join the alliance. At the conference, Levison recounted much of what's happened over the last few months (with quite a bit of humor), joking about how he tried to be "nice" in giving the feds Lavabit's private keys printed out, by noting that he included line numbers to help (leaving unsaid that this would make OCR'ing the keys even more difficult). He also admitted that giving them the paper version was really just a way to buy time to shut down Lavabit.

Former NSA and CIA director Michael Hayden stood on the pulpit of a church across from the White House on Sunday and declared Gmail the preferred online service of terrorists. As part of an adult education forum at St. John’s Episcopal Church, Hayden gave a wide ranging speech on "the tension between security and liberty." During the speech, he specifically defended Section 702 of the Foreign Surveillance Intelligence Act (FISA), which provides the legal basis for the PRISM program. In doing so, Hayden claimed "Gmail is the preferred Internet service provider of terrorists worldwide," presumably meaning online service rather than the actual provider of Internet service. He added: "I don't think you're going to see that in a Google commercial, but it's free, it's ubiquitous, so of course it is."

When the National Security Agency’s bulk collection of records about Americans’ emails came to light in 2013, the government conceded the program’s existence but said it had shut down the effort in December 2011 for “operational and resource reasons.” While that particular secret program stopped, newly disclosed documents show that the N.S.A. had found a way to create a functional equivalent. The shift has permitted the agency to continue analyzing social links revealed by Americans’ email patterns, but without collecting the data in bulk from American telecommunications companies — and with less oversight by the Foreign Intelligence Surveillance Court.

The disclosure comes as a sister program that collects Americans’ phone records in bulk is set to end this month. Under a law enacted in June, known as the U.S.A. Freedom Act, the program will be replaced with a system in which the N.S.A. can still gain access to the data to hunt for associates of terrorism suspects, but the bulk logs will stay in the hands of phone companies.The newly disclosed information about the email records program is contained in a report by the N.S.A.’s inspector general that was obtained by The New York Times through a lawsuit under the Freedom of Information Act. One passage lists four reasons that the N.S.A. decided to end the email program and purge previously collected data. Three were redacted, but the fourth was uncensored. It said that “other authorities can satisfy certain foreign intelligence requirements” that the bulk email records program “had been designed to meet.”The report explained that there were two other legal ways to get such data. One was the collection of bulk data that had been gathered in other countries, where the N.S.A.’s activities are largely not subject to regulation by the Foreign Intelligence Surveillance Act and oversight by the intelligence court. Because of the way the Internet operates, domestic data is often found on fiber optic cables abroad.

The N.S.A. had long barred analysts from using Americans’ data that had been swept up abroad, but in November 2010 it changed that rule, documents leaked by Edward J. Snowden have shown. The inspector general report cited that change to the N.S.A.’s internal procedures.The other replacement source for the data was collection under the FISA Amendments Act of 2008, which permits warrantless surveillance on domestic soil that targets specific noncitizens abroad, including their new or stored emails to or from Americans.“Thus,” the report said, these two sources “assist in the identification of terrorists communicating with individuals in the United States, which addresses one of the original reasons for establishing” the bulk email records program.

Current and former FBI officials have launched a media counter-offensive to engage head to head with the Clinton media machine and to throw off the shackles the Loretta Lynch Justice Department has used to stymie their multiple investigations into the Clinton pay-to-play network. Over the past weekend, former FBI Assistant Director and current CNN Senior Law Enforcement Analyst Tom Fuentes told viewers that “the FBI has an intensive investigation ongoing into the Clinton Foundation.” He said he had received this information from “senior officials” at the FBI, “several of them, in and out of the Bureau.” (See video clip from CNN below.) That information was further supported by an in-depth article last evening in the Wall Street Journal by Devlin Barrett. According to Barrett, the “probe of the foundation began more than a year ago to determine whether financial crimes or influence peddling occurred related to the charity.” Barrett’s article suggests that the Justice Department, which oversees the FBI, has attempted to circumvent the investigation. The new revelations lead to the appearance of wrongdoing on the part of U.S. Attorney General Loretta Lynch for secretly meeting with Bill Clinton on her plane on the tarmac of Phoenix Sky Harbor International Airport on the evening of June 28 of this year. Not only was Bill Clinton’s wife under an FBI investigation at the time over her use of a private email server in the basement of her New York home over which Top Secret material was transmitted while she was Secretary of State but his own charitable foundation was also under investigation, a fact that was unknown at the time to the public and the media.

The reports leaking out of the FBI over the weekend came on the heels of FBI Director James Comey sending a letter to members of Congress on Friday acknowledging that the investigation into the Hillary Clinton email server was not closed as he had previously testified to Congress, but had been reopened as a result of “pertinent” emails turning up. According to multiple media sources, those emails were found on the laptop of Anthony Weiner, estranged husband of Hillary Clinton’s longtime aide, Huma Abedin. Weiner was forced to resign from Congress in 2011 over a sexting scandal with more sexting scandals to follow. Early this month, on October 3, the FBI raided Weiner’s apartment in New York with a search warrant in hand and seized multiple electronic devices. At least one of those devices had been used by both Weiner and Abedin to send email messages. The search warrant had been obtained following a detailed report that had appeared in the Daily Mail newspaper in the U.K. in September, showing sordid, sexual emails that Weiner had allegedly sent to a 15-year old girl in North Carolina. According to the content of the published emails, Weiner was aware that the girl was underage. While examining the hard drive of a laptop seized from Weiner, the FBI found metadata showing that thousands of the emails were exchanged between the private server located in Hillary Clinton’s basement in Chappaqua, New York. Since the FBI had only a search warrant to examine emails and documents related to the Weiner matter, it was not able to read the other emails but simply tallied up the metadata linked to the private server in Chappaqua to show how many potentially new emails might be in the trove on the laptop. (It is also possible that the FBI agents could have used a program to compare subject headings against those emails previously turned over by Abedin to make their case that Abedin had perjured herself when she testified to them that she had turned over all of her work-related emails from her time as Deputy Chief of Staff for Operations when Hillary Clinton served as Secretary of State.)

The FBI officials briefed FBI Director James Comey last Thursday on the new trove of emails and he went public with his letter to Congress the next day – leading the Clinton camp to immediately launch a media assault on his motives. As if all of this wasn’t enough for the public to digest over the weekend, yesterday morning James Kallstrom, the former Assistant Director in Charge of the FBI’s New York Division and 25-year veteran of the FBI, went on the John Catsimatidis radio program to call the Clintons a “crime family.” (You can hear the full program here.) Kallstrom’s comments were further amplified when the New York Daily News reported on his criticisms that the FBI had been hamstrung from conducting a real investigation. The Daily News quoted Kallstrom from the radio program as follows: “ ‘This investigation was never a real investigation, they never had grand jury empaneled…This investigation was without the ability to serve subpoenas, serve search warrants, and obtain evidence … It was just ludicrous.’ ” On Wednesday of last week, just two days before FBI Director Comey issued his public letter to Congress, WikiLeaks released a devastating memo from Douglas Band, at the time a key figure at the Clinton Foundation and President of his own corporate consulting firm, Teneo. The memo was rife with suggestions of pay-to-play and personal enrichment schemes at the Clinton Foundation. The same day, Washington Post reporters Rosalind Helderman and Tom Hamburger reported the story, including the following details:

Director of National Intelligence James Clapper has won the infamous Rosemary Award for worst open government performance in 2013, according to the citation published today by the National Security Archive at www.nsarchive.org. Despite heavy competition, Clapper's "No, sir" lie to Senator Ron Wyden's question: "Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?" sealed his receipt of the dubious achievement award, which cites the vastly excessive secrecy of the entire U.S. surveillance establishment. The Rosemary Award citation leads with what Clapper later called the "least untruthful" answer possible to congressional questions about the secret bulk collection of Americans' phone call data. It further cites other Clapper claims later proved false, such as his 2012 statement that "we don't hold data on U.S. citizens." But the Award also recognizes Clapper's fellow secrecy fetishists and enablers, including:

Gen. Keith Alexander, director of the NSA, for multiple Rose Mary Woods-type stretches, such as (1) claiming that the secret bulk collection prevented 54 terrorist plots against the U.S. when the actual number, according to the congressionally-established Privacy and Civil Liberties Oversight Board (PCLOB) investigation (pp. 145-153), is zero; (2) his 2009 declaration to the wiretap court that multiple NSA violations of the court's orders arose from differences over "terminology," an explanation which the chief judge said "strains credulity;" and (3) public statements by the NSA about its programs that had to be taken down from its website for inaccuracies (see Documents 78, 85, 87 in The Snowden Affair), along with public statements by other top NSA officials now known to be untrue (see "Remarks of Rajesh De," NSA General Counsel, Document 53 in The Snowden Affair).

Robert Mueller, former FBI director, for suggesting (as have Gen. Alexander and many others) that the secret bulk collection program might have been able to prevent the 9/11 attacks, when the 9/11 Commission found explicitly the problem was not lack of data points, but failing to connect the many dots the intelligence community already had about the would-be hijackers living in San Diego. The National Security Division lawyers at the Justice Department, for misleading their own Solicitor General (Donald Verrilli) who then misled (inadvertently) the U.S. Supreme Court over whether Justice let defendants know that bulk collection had contributed to their prosecutions. The same National Security Division lawyers who swore under oath in the Electronic Frontier Foundation's Freedom of Information Act lawsuit for a key wiretap court opinion that the entire text of the opinion was appropriately classified Top Secret/Sensitive Compartmented Information (release of which would cause "exceptionally grave damage" to U.S. national security). Only after the Edward Snowden leaks and the embarrassed governmental declassification of the opinion did we find that one key part of the opinion's text simply reproduced the actual language of the 4th Amendment to the U.S. Constitution, and the only "grave damage" was to the government's false claims.

Britain's spy agency GCHQ has secretly gained access to the network of cables which carry the world's phone calls and internet traffic and has started to process vast streams of sensitive personal information which it is sharing with its American partner, the National Security Agency (NSA).The sheer scale of the agency's ambition is reflected in the titles of its two principal components: Mastering the Internet and Global Telecoms Exploitation, aimed at scooping up as much online and telephone traffic as possible. This is all being carried out without any form of public acknowledgement or debate.One key innovation has been GCHQ's ability to tap into and store huge volumes of data drawn from fibre-optic cables for up to 30 days so that it can be sifted and analysed. That operation, codenamed Tempora, has been running for some 18 months.

GCHQ and the NSA are consequently able to access and process vast quantities of communications between entirely innocent people, as well as targeted suspects.This includes recordings of phone calls, the content of email messages, entries on Facebook and the history of any internet user's access to websites – all of which is deemed legal, even though the warrant system was supposed to limit interception to a specified range of targets.The existence of the programme has been disclosed in documents shown to the Guardian by the NSA whistleblower Edward Snowden as part of his attempt to expose what he has called "the largest programme of suspicionless surveillance in human history"."It's not just a US problem. The UK has a huge dog in this fight," Snowden told the Guardian. "They [GCHQ] are worse than the US."

However, on Friday a source with knowledge of intelligence argued that the data was collected legally under a system of safeguards, and had provided material that had led to significant breakthroughs in detecting and preventing serious crime.Britain's technical capacity to tap into the cables that carry the world's communications – referred to in the documents as special source exploitation – has made GCHQ an intelligence superpower.By 2010, two years after the project was first trialled, it was able to boast it had the "biggest internet access" of any member of the Five Eyes electronic eavesdropping alliance, comprising the US, UK, Canada, Australia and New Zealand.UK officials could also claim GCHQ "produces larger amounts of metadata than NSA". (Metadata describes basic information on who has been contacting whom, without detailing the content.)By May last year 300 analysts from GCHQ, and 250 from the NSA, had been assigned to sift through the flood of data.The Americans were given guidelines for its use, but were told in legal briefings by GCHQ lawyers: "We have a light oversight regime compared with the US".

Newly declassified court documents indicate that the National Security Agency shared its trove of American bulk email and internet data with other government agencies in violation of specific court-ordered procedures to protect Americans’ privacy. The dissemination of the sensitive data transgressed both the NSA’s affirmations to the secret surveillance court about the extent of the access it provided, and prompted incensed Fisa court judges to question both the NSA’s truthfulness and the value of the now-cancelled program to counter-terrorism. While the NSA over the past several months has portrayed its previous violations of Fisa court orders as “technical” violations or inadvertent errors, the oversharing of internet data is described in the documents as apparent widespread and unexplained procedural violations. “NSA’s record of compliance with these rules has been poor,” wrote judge John Bates in an opinion released on Monday night in which the date is redacted.

“Most notably, NSA generally disregarded the special rules for disseminating United States person information outside of NSA until it was ordered to report such disseminations and to certify to the [Fisa court] that the required approval had been obtained.” In addition to improperly permitting access to the email and internet data – intended to include information such as the “to” “from” and “BCC” lines of an email – Bates found that the NSA engaged in “systemic overcollection”, suggesting that content of Americans’ communications was collected as well.

The court had required the NSA to comply with a longstanding internal procedure for protecting Americans’ sensitive information prior to sharing the data internally within NSA, known as United States Signals Intelligence Directive 18 (USSID 18) and also declassified on Monday night; and additionally required a senior NSA official to determine that any material shared outside the powerful surveillance agency was related to counter-terrorism. Yet in a separate Fisa court document, the current presiding judge, Reggie Walton, blasted the government’s secret declaration that it followed USSID 18 “rather than specifically requiring that the narrower dissemination provision set forth in the Court’s orders in this matter be strictly adhered to”. Walton wrote: “The court understands this to mean that the NSA likely has disseminated US person information derived from the [email and internet bulk] metadata outside NSA without a prior determination from the NSA official designated in the court’s orders that the information is related to counter-terrorism information and is necessary to understand the counter-terrorism information or assess its importance.”

The Justice Department has released a newly declassified version of a May 2004 legal memo approving the National Security Agency’s Stellarwind program, a set of warrantless surveillance and data collection activities that President George W. Bush secretly authorized after the terrorist attacks of Sept. 11, 2001. But questions about the program remain.A more heavily redacted version of the memo had been released in 2011 as part of Freedom of Information Act lawsuits by the American Civil Liberties Union and the Electronic Privacy Information Center. The new version includes previously censored references to the existence of the data collection related to Americans’ phone calls and emails.

The Obama administration voluntarily reprocessed the memo from Jack Goldsmith, the head of the Justice Department’s Office of Legal Counsel, in light of the fact that it had declassified the existence of the bulk phone and email data programs last year after leaks by Edward J. Snowden, a former N.S.A. contractor.The fuller release adds to the public record of an important historical episode. However, the government continued to redact crucial portions of the memo that would answer a primary remaining question about the history of Stellarwind: What prompted the Justice Department to conclude in early 2004 that one aspect of the program, which collected records about Americans’ emails in bulk, was illegal — even though it permitted other aspects, like warrantless wiretapping and the bulk collection of Americans’ phone records, to continue?“They have continued to keep redacted something very significant,” said Jameel Jaffer, a lawyer with the A.C.L.U.

The Justice Department’s conclusion that the email metadata program was illegal led to a March 2004 confrontation between White House and department officials in the hospital room of Attorney General John Ashcroft, after which nearly the entire top leadership of the department threatened to resign, prompting President Bush to agree to changes.

One system builds profiles showing people’s web browsing histories. Another analyzes instant messenger communications, emails, Skype calls, text messages, cell phone locations, and social media interactions. Separate programs were built to keep tabs on “suspicious” Google searches and usage of Google Maps. The surveillance is underpinned by an opaque legal regime that has authorized GCHQ to sift through huge archives of metadata about the private phone calls, emails and Internet browsing logs of Brits, Americans, and any other citizens  all without a court order or judicial warrant.

The power of KARMA POLICE was illustrated in 2009, when GCHQ launched a top-secret operation to collect intelligence about people using the Internet to listen to radio shows. The agency used a sample of nearly 7 million metadata records, gathered over a period of three months, to observe the listening habits of more than 200,000 people across 185 countries, including the U.S., the U.K., Ireland, Canada, Mexico, Spain, the Netherlands, France, and Germany.

GCHQ’s documents indicate that the plans for KARMA POLICE were drawn up between 2007 and 2008. The system was designed to provide the agency with “either (a) a web browsing profile for every visible user on the Internet, or (b) a user profile for every visible website on the Internet.” The origin of the surveillance system’s name is not discussed in the documents. But KARMA POLICE is also the name of a popular song released in 1997 by the Grammy Award-winning British band Radiohead, suggesting the spies may have been fans. A verse repeated throughout the hit song includes the lyric, “This is what you’ll get, when you mess with us.”

The National Security Agency has a secret backdoor into its vast databases under a legal authority enabling it to search for US citizens' email and phone calls without a warrant, according to a top-secret document passed to the Guardian by Edward Snowden.The previously undisclosed rule change allows NSA operatives to hunt for individual Americans' communications using their name or other identifying information. Senator Ron Wyden told the Guardian that the law provides the NSA with a loophole potentially allowing "warrantless searches for the phone calls or emails of law-abiding Americans".The authority, approved in 2011, appears to contrast with repeated assurances from Barack Obama and senior intelligence officials to both Congress and the American public that the privacy of US citizens is protected from the NSA's dragnet surveillance programs.

The intelligence data is being gathered under Section 702 of the of the Fisa Amendments Act (FAA), which gives the NSA authority to target without warrant the communications of foreign targets, who must be non-US citizens and outside the US at the point of collection.The communications of Americans in direct contact with foreign targets can also be collected without a warrant, and the intelligence agencies acknowledge that purely domestic communications can also be inadvertently swept into its databases. That process is known as "incidental collection" in surveillance parlance.But this is the first evidence that the NSA has permission to search those databases for specific US individuals' communications.

A secret glossary document provided to operatives in the NSA's Special Source Operations division – which runs the Prism program and large-scale cable intercepts through corporate partnerships with technology companies – details an update to the "minimization" procedures that govern how the agency must handle the communications of US persons. That group is defined as both American citizens and foreigners located in the US."While the FAA 702 minimization procedures approved on 3 October 2011 now allow for use of certain United States person names and identifiers as query terms when reviewing collected FAA 702 data," the glossary states, "analysts may NOT/NOT [not repeat not] implement any USP [US persons] queries until an effective oversight process has been developed by NSA and agreed to by DOJ/ODNI [Office of the Director of National Intelligence]."The term "identifiers" is NSA jargon for information relating to an individual, such as telephone number, email address, IP address and username as well as their name.The document – which is undated, though metadata suggests this version was last updated in June 2012 – does not say whether the oversight process it mentions has been established or whether any searches against US person names have taken place.

Were Israeli companies Verint and Narus the ones that collected information from the U.S. communications network for the National Security Agency? The question arises amid controversy over revelations that the NSA has been collecting the phone records of hundreds of millions of Americans every day, creating a database through which it can learn whether terror suspects have been in contact with people in the United States. It also was disclosed this week that the NSA has been gathering all Internet usage - audio, video, photographs, emails and searches - from nine major U.S. Internet providers, including Microsoft and Google, in hopes of detecting suspicious behavior that begins overseas.

According to an article in the American technology magazine "Wired" from April 2012, two Israeli companies – which the magazine describes as having close connections to the Israeli security community – conduct bugging and wiretapping for the NSA. Verint, which took over its parent company Comverse Technology earlier this year, is responsible for tapping the communication lines of the American telephone giant Verizon, according to a past Verizon employee sited by James Bamford in Wired. Neither Verint nor Verizon commented on the matter.

Natus, which was acquired in 2010 by the American company Boeing, supplied the software and hardware used at AT&T wiretapping rooms, according to whistleblower Mark Klein, who revealed the information in 2004. Klein, a past technician at AT&T who filed a suit against the company for spying on its customers, revealed a "secret room" in the company's San Fransisco office, where the NSA collected data on American citizens' telephone calls and Internet surfing. Klein's claims were reinforced by former NSA employee Thomas Drake who testified that the agency uses a program produced by Narus to save the personal electrical communications of AT&T customers.  Both Verint and Narus have ties to the Israeli intelligence agency and the Israel Defense Forces intelligence-gathering unit 8200. Hanan Gefen, a former commander of the 8200 unit, told Forbes magazine in 2007 that Comverse's technology, which was formerly the parent company of Verint and merged with it this year, was directly influenced by the technology of 8200. Ori Cohen, one of the founders of Narus, told Fortune magazine in 2001 that his partners had done technology work for the Israeli intelligence.

Senior figures inside British intelligence have been alarmed by GCHQ's secret decision to tap into transatlantic cables in order to engage in the bulk interception of phone calls and internet traffic.According to one source who has been directly involved in GCHQ operations, concerns were expressed when the project was being discussed internally in 2008: "We felt we were starting to overstep the mark with some of it. People from MI5 were complaining that they were going too far from a civil liberties perspective … We all had reservations about it, because we all thought: 'If this was used against us, we wouldn't stand a chance'."The Guardian revealed on Friday that GCHQ has placed more than 200 probes on transatlantic cables and is processing 600m "telephone events" a day as well as up to 39m gigabytes of internet traffic. Using a programme codenamed Tempora, it can store and analyse voice recordings, the content of emails, entries on Facebook, the use of websites as well as the "metadata" which records who has contacted who. The programme is shared with GCHQ's American partner, the National Security Agency.

Internet traffic is also liable to be routed internationally even if the message is exchanged between two people within the UK. "At one point, I was told that we were getting 85% of all UK domestic traffic – voice, internet, all of it – via these international cables."

The source claimed that even the conventional warrant system has been distorted – whereas police used to ask for a warrant before intercepting a target's communications, they will now ask GCHQ to intercept the target's communications and then use that information to seek a warrant.There is a particular concern that the programme allows GCHQ to break the boundary which stopped it engaging in the bulk interception of internal UK communications. The Ripa requirement that one end of a communication must be outside the UK was a significant restriction when it was applied to phone calls using satellites, but it is no longer effective in the world of fibre-optic cables. "The point is that this is an island," the source said. "Everything comes and goes – nearly everything – down fibre-optic cables. You make a mobile phone call, it goes to a mast and then down into a fibre-optic cable, under the ground and away. And even if the call is UK to UK, it's very likely – because of the way the system is structured – to go out of the UK and come back in through these fibre-optic channels."

Telephone and Internet metadata are protected by law under the aforementioned “pen register” statute.  That statute says that “no person may install or use a pen register or a trap and trace device without first obtaining a court order under section 3123 of this title or under the Foreign Intelligence Surveillance Act of 1978 (50 U.S.C. 1801 et seq.).” Pen registers and trap and trace devices collect dialing, routing, signaling or addressing information. Violation of the statute is a criminal misdemeanor.