Group items tagged

A huge volume of the Internet data GCHQ collects flows directly into a massive repository named Black Hole, which is at the core of the agency’s online spying operations, storing raw logs of intercepted material before it has been subject to analysis. Black Hole contains data collected by GCHQ as part of bulk “unselected” surveillance, meaning it is not focused on particular “selected” targets and instead includes troves of data indiscriminately swept up about ordinary people’s online activities. Between August 2007 and March 2009, GCHQ documents say that Black Hole was used to store more than 1.1 trillion “events” — a term the agency uses to refer to metadata records — with about 10 billion new entries added every day. As of March 2009, the largest slice of data Black Hole held — 41 percent — was about people’s Internet browsing histories. The rest included a combination of email and instant messenger records, details about search engine queries, information about social media activity, logs related to hacking operations, and data on people’s use of tools to browse the Internet anonymously.

Throughout this period, as smartphone sales started to boom, the frequency of people’s Internet use was steadily increasing. In tandem, British spies were working frantically to bolster their spying capabilities, with plans afoot to expand the size of Black Hole and other repositories to handle an avalanche of new data. By 2010, according to the documents, GCHQ was logging 30 billion metadata records per day. By 2012, collection had increased to 50 billion per day, and work was underway to double capacity to 100 billion. The agency was developing “unprecedented” techniques to perform what it called “population-scale” data mining, monitoring all communications across entire countries in an effort to detect patterns or behaviors deemed suspicious. It was creating what it said would be, by 2013, “the world’s biggest” surveillance engine “to run cyber operations and to access better, more valued data for customers to make a real world difference.”

A document from the GCHQ target analysis center (GTAC) shows the Black Hole repository’s structure.

The data is searched by GCHQ analysts in a hunt for behavior online that could be connected to terrorism or other criminal activity. But it has also served a broader and more controversial purpose — helping the agency hack into European companies’ computer networks. In the lead up to its secret mission targeting Netherlands-based Gemalto, the largest SIM card manufacturer in the world, GCHQ used MUTANT BROTH in an effort to identify the company’s employees so it could hack into their computers. The system helped the agency analyze intercepted Facebook cookies it believed were associated with Gemalto staff located at offices in France and Poland. GCHQ later successfully infiltrated Gemalto’s internal networks, stealing encryption keys produced by the company that protect the privacy of cell phone communications.

Similarly, MUTANT BROTH proved integral to GCHQ’s hack of Belgian telecommunications provider Belgacom. The agency entered IP addresses associated with Belgacom into MUTANT BROTH to uncover information about the company’s employees. Cookies associated with the IPs revealed the Google, Yahoo, and LinkedIn accounts of three Belgacom engineers, whose computers were then targeted by the agency and infected with malware. The hacking operation resulted in GCHQ gaining deep access into the most sensitive parts of Belgacom’s internal systems, granting British spies the ability to intercept communications passing through the company’s networks.

In March, a U.K. parliamentary committee published the findings of an 18-month review of GCHQ’s operations and called for an overhaul of the laws that regulate the spying. The committee raised concerns about the agency gathering what it described as “bulk personal datasets” being held about “a wide range of people.” However, it censored the section of the report describing what these “datasets” contained, despite acknowledging that they “may be highly intrusive.” The Snowden documents shine light on some of the core GCHQ bulk data-gathering programs that the committee was likely referring to — pulling back the veil of secrecy that has shielded some of the agency’s most controversial surveillance operations from public scrutiny. KARMA POLICE and MUTANT BROTH are among the key bulk collection systems. But they do not operate in isolation — and the scope of GCHQ’s spying extends far beyond them.

The agency operates a bewildering array of other eavesdropping systems, each serving its own specific purpose and designated a unique code name, such as: SOCIAL ANTHROPOID, which is used to analyze metadata on emails, instant messenger chats, social media connections and conversations, plus “telephony” metadata about phone calls, cell phone locations, text and multimedia messages; MEMORY HOLE, which logs queries entered into search engines and associates each search with an IP address; MARBLED GECKO, which sifts through details about searches people have entered into Google Maps and Google Earth; and INFINITE MONKEYS, which analyzes data about the usage of online bulletin boards and forums. GCHQ has other programs that it uses to analyze the content of intercepted communications, such as the full written body of emails and the audio of phone calls. One of the most important content collection capabilities is TEMPORA, which mines vast amounts of emails, instant messages, voice calls and other communications and makes them accessible through a Google-style search tool named XKEYSCORE.

As of September 2012, TEMPORA was collecting “more than 40 billion pieces of content a day” and it was being used to spy on people across Europe, the Middle East, and North Africa, according to a top-secret memo outlining the scope of the program. The existence of TEMPORA was first revealed by The Guardian in June 2013. To analyze all of the communications it intercepts and to build a profile of the individuals it is monitoring, GCHQ uses a variety of different tools that can pull together all of the relevant information and make it accessible through a single interface. SAMUEL PEPYS is one such tool, built by the British spies to analyze both the content and metadata of emails, browsing sessions, and instant messages as they are being intercepted in real time. One screenshot of SAMUEL PEPYS in action shows the agency using it to monitor an individual in Sweden who visited a page about GCHQ on the U.S.-based anti-secrecy website Cryptome.

Partly due to the U.K.’s geographic location — situated between the United States and the western edge of continental Europe — a large amount of the world’s Internet traffic passes through its territory across international data cables. In 2010, GCHQ noted that what amounted to “25 percent of all Internet traffic” was transiting the U.K. through some 1,600 different cables. The agency said that it could “survey the majority of the 1,600” and “select the most valuable to switch into our processing systems.”

According to Joss Wright, a research fellow at the University of Oxford’s Internet Institute, tapping into the cables allows GCHQ to monitor a large portion of foreign communications. But the cables also transport masses of wholly domestic British emails and online chats, because when anyone in the U.K. sends an email or visits a website, their computer will routinely send and receive data from servers that are located overseas. “I could send a message from my computer here [in England] to my wife’s computer in the next room and on its way it could go through the U.S., France, and other countries,” Wright says. “That’s just the way the Internet is designed.” In other words, Wright adds, that means “a lot” of British data and communications transit across international cables daily, and are liable to be swept into GCHQ’s databases.

A map from a classified GCHQ presentation about intercepting communications from undersea cables. GCHQ is authorized to conduct dragnet surveillance of the international data cables through so-called external warrants that are signed off by a government minister. The external warrants permit the agency to monitor communications in foreign countries as well as British citizens’ international calls and emails — for example, a call from Islamabad to London. They prohibit GCHQ from reading or listening to the content of “internal” U.K. to U.K. emails and phone calls, which are supposed to be filtered out from GCHQ’s systems if they are inadvertently intercepted unless additional authorization is granted to scrutinize them. However, the same rules do not apply to metadata. A little-known loophole in the law allows GCHQ to use external warrants to collect and analyze bulk metadata about the emails, phone calls, and Internet browsing activities of British people, citizens of closely allied countries, and others, regardless of whether the data is derived from domestic U.K. to U.K. communications and browsing sessions or otherwise. In March, the existence of this loophole was quietly acknowledged by the U.K. parliamentary committee’s surveillance review, which stated in a section of its report that “special protection and additional safeguards” did not apply to metadata swept up using external warrants and that domestic British metadata could therefore be lawfully “returned as a result of searches” conducted by GCHQ.

Perhaps unsurprisingly, GCHQ appears to have readily exploited this obscure legal technicality. Secret policy guidance papers issued to the agency’s analysts instruct them that they can sift through huge troves of indiscriminately collected metadata records to spy on anyone regardless of their nationality. The guidance makes clear that there is no exemption or extra privacy protection for British people or citizens from countries that are members of the Five Eyes, a surveillance alliance that the U.K. is part of alongside the U.S., Canada, Australia, and New Zealand. “If you are searching a purely Events only database such as MUTANT BROTH, the issue of location does not occur,” states one internal GCHQ policy document, which is marked with a “last modified” date of July 2012. The document adds that analysts are free to search the databases for British metadata “without further authorization” by inputing a U.K. “selector,” meaning a unique identifier such as a person’s email or IP address, username, or phone number. Authorization is “not needed for individuals in the U.K.,” another GCHQ document explains, because metadata has been judged “less intrusive than communications content.” All the spies are required to do to mine the metadata troves is write a short “justification” or “reason” for each search they conduct and then click a button on their computer screen.

Intelligence GCHQ collects on British persons of interest is shared with domestic security agency MI5, which usually takes the lead on spying operations within the U.K. MI5 conducts its own extensive domestic surveillance as part of a program called DIGINT (digital intelligence).

GCHQ’s documents suggest that it typically retains metadata for periods of between 30 days to six months. It stores the content of communications for a shorter period of time, varying between three to 30 days. The retention periods can be extended if deemed necessary for “cyber defense.” One secret policy paper dated from January 2010 lists the wide range of information the agency classes as metadata — including location data that could be used to track your movements, your email, instant messenger, and social networking “buddy lists,” logs showing who you have communicated with by phone or email, the passwords you use to access “communications services” (such as an email account), and information about websites you have viewed.

Records showing the full website addresses you have visited — for instance, www.gchq.gov.uk/what_we_do — are treated as content. But the first part of an address you have visited — for instance, www.gchq.gov.uk — is treated as metadata. In isolation, a single metadata record of a phone call, email, or website visit may not reveal much about a person’s private life, according to Ethan Zuckerman, director of Massachusetts Institute of Technology’s Center for Civic Media. But if accumulated and analyzed over a period of weeks or months, these details would be “extremely personal,” he told The Intercept, because they could reveal a person’s movements, habits, religious beliefs, political views, relationships, and even sexual preferences. For Zuckerman, who has studied the social and political ramifications of surveillance, the most concerning aspect of large-scale government data collection is that it can be “corrosive towards democracy” — leading to a chilling effect on freedom of expression and communication. “Once we know there’s a reasonable chance that we are being watched in one fashion or another it’s hard for that not to have a ‘panopticon effect,’” he said, “where we think and behave differently based on the assumption that people may be watching and paying attention to what we are doing.”

When compared to surveillance rules in place in the U.S., GCHQ notes in one document that the U.K. has “a light oversight regime.” The more lax British spying regulations are reflected in secret internal rules that highlight greater restrictions on how NSA databases can be accessed. The NSA’s troves can be searched for data on British citizens, one document states, but they cannot be mined for information about Americans or other citizens from countries in the Five Eyes alliance. No such constraints are placed on GCHQ’s own databases, which can be sifted for records on the phone calls, emails, and Internet usage of Brits, Americans, and citizens from any other country. The scope of GCHQ’s surveillance powers explain in part why Snowden told The Guardian in June 2013 that U.K. surveillance is “worse than the U.S.” In an interview with Der Spiegel in July 2013, Snowden added that British Internet cables were “radioactive” and joked: “Even the Queen’s selfies to the pool boy get logged.”

In recent years, the biggest barrier to GCHQ’s mass collection of data does not appear to have come in the form of legal or policy restrictions. Rather, it is the increased use of encryption technology that protects the privacy of communications that has posed the biggest potential hindrance to the agency’s activities. “The spread of encryption … threatens our ability to do effective target discovery/development,” says a top-secret report co-authored by an official from the British agency and an NSA employee in 2011. “Pertinent metadata events will be locked within the encrypted channels and difficult, if not impossible, to prise out,” the report says, adding that the agencies were working on a plan that would “(hopefully) allow our Internet Exploitation strategy to prevail.”

When it came to judging the necessity and proportionality of what they were allowed to look for, would-be American users were told it was "your call".The Guardian understands that a total of 850,000 NSA employees and US private contractors with top secret clearance had access to GCHQ databases.

For the 2 billion users of the world wide web, Tempora represents a window on to their everyday lives, sucking up every form of communication from the fibre-optic cables that ring the world.The NSA has meanwhile opened a second window, in the form of the Prism operation, revealed earlier this month by the Guardian, from which it secured access to the internal systems of global companies that service the internet.The GCHQ mass tapping operation has been built up over five years by attaching intercept probes to transatlantic fibre-optic cables where they land on British shores carrying data to western Europe from telephone exchanges and internet servers in north America.This was done under secret agreements with commercial companies, described in one document as "intercept partners".The papers seen by the Guardian suggest some companies have been paid for the cost of their co-operation and GCHQ went to great lengths to keep their names secret. They were assigned "sensitive relationship teams" and staff were urged in one internal guidance paper to disguise the origin of "special source" material in their reports for fear that the role of the companies as intercept partners would cause "high-level political fallout".

The GCHQ documents that the Guardian has seen illustrate a constant effort to build up storage capacity at the stations at Cheltenham, Bude and at one overseas location, as well a search for ways to maintain the agency's comparative advantage as the world's leading communications companies increasingly route their cables through Asia to cut costs. Meanwhile, technical work is ongoing to expand GCHQ's capacity to ingest data from new super cables carrying data at 100 gigabits a second. As one training slide told new users: "You are in an enviable position – have fun and make the most of it."

The categories of material have included fraud, drug trafficking and terrorism, but the criteria at any one time are secret and are not subject to any public debate. GCHQ's compliance with the certificates is audited by the agency itself, but the results of those audits are also secret.An indication of how broad the dragnet can be was laid bare in advice from GCHQ's lawyers, who said it would be impossible to list the total number of people targeted because "this would be an infinite list which we couldn't manage".There is an investigatory powers tribunal to look into complaints that the data gathered by GCHQ has been improperly used, but the agency reassured NSA analysts in the early days of the programme, in 2009: "So far they have always found in our favour".

Historically, the spy agencies have intercepted international communications by focusing on microwave towers and satellites. The NSA's intercept station at Menwith Hill in North Yorkshire played a leading role in this. One internal document quotes the head of the NSA, Lieutenant General Keith Alexander, on a visit to Menwith Hill in June 2008, asking: "Why can't we collect all the signals all the time? Sounds like a good summer project for Menwith."By then, however, satellite interception accounted for only a small part of the network traffic. Most of it now travels on fibre-optic cables, and the UK's position on the western edge of Europe gave it natural access to cables emerging from the Atlantic.

The processing centres apply a series of sophisticated computer programmes in order to filter the material through what is known as MVR – massive volume reduction. The first filter immediately rejects high-volume, low-value traffic, such as peer-to-peer downloads, which reduces the volume by about 30%. Others pull out packets of information relating to "selectors" – search terms including subjects, phone numbers and email addresses of interest. Some 40,000 of these were chosen by GCHQ and 31,000 by the NSA. Most of the information extracted is "content", such as recordings of phone calls or the substance of email messages. The rest is metadata.

"The criteria are security, terror, organised crime. And economic well-being. There's an auditing process to go back through the logs and see if it was justified or not. The vast majority of the data is discarded without being looked at … we simply don't have the resources."However, the legitimacy of the operation is in doubt. According to GCHQ's legal advice, it was given the go-ahead by applying old law to new technology. The 2000 Regulation of Investigatory Powers Act (Ripa) requires the tapping of defined targets to be authorised by a warrant signed by the home secretary or foreign secretary.However, an obscure clause allows the foreign secretary to sign a certificate for the interception of broad categories of material, as long as one end of the monitored communications is abroad. But the nature of modern fibre-optic communications means that a proportion of internal UK traffic is relayed abroad and then returns through the cables.

British spy agency collects and stores vast quantities of global email messages, Facebook posts, internet histories and calls, and shares them with NSA, latest documents from Edward Snowden reveal

In reality, numerous telecoms companies were doing much more than that, as disclosed in a secret document prepared in 2009 by a joint working group of GCHQ, MI5 and MI6.Their report contended that allowing intercepts as evidence could damage relationships with "Communications Service Providers" (CSPs).In an extended excerpt of "the classified version" of a review prepared for the Privy Council, a formal body of advisers made up of current and former cabinet ministers, the document sets out the real nature of the relationship between telecoms firms and the UK government."Under RIPA [the Regulation of Investigatory Powers Act 2000], CSPs in the UK may be required to provide, at public expense, an adequate interception capability on their networks," it states. "In practice all significant providers do provide such a capability. But in many cases their assistance – while in conformity with the law – goes well beyond what it requires."

GCHQ's internet surveillance programme is the subject of a challenge in the European court of human rights, mounted by three privacy advocacy groups. The Open Rights Group, English PEN and Big Brother Watch argue the "unchecked surveillance" of Tempora is a challenge to the right to privacy, as set out in the European convention on human rights.That the Tempora programme appears to rely at least in part on voluntary co-operation of telecoms firms could become a major factor in that ongoing case. The revelation could also reignite the long-running debate over allowing intercept evidence in court.GCHQ's submission goes on to set out why its relationships with telecoms companies go further than what can be legally compelled under current law. It says that in the internet era, companies wishing to avoid being legally mandated to assist UK intelligence agencies would often be able to do so "at little cost or risk to their operations" by moving "some or all" of their communications services overseas.

As a result, "it has been necessary to enter into agreements with both UK-based and offshore providers for them to afford the UK agencies access, with appropriate legal authorisation, to the communications they carry outside the UK".The submission to ministers does not set out which overseas firms have entered into voluntary relationships with the UK, or even in which countries they operate, though documents detailing the Tempora programme made it clear the UK's interception capabilities relied on taps located both on UK soil and overseas.There is no indication as to whether the governments of the countries in which deals with companies have been struck would be aware of the GCHQ cable taps.

Evidence that telecoms firms and GCHQ are engaging in mass interception overseas could stoke an ongoing diplomatic row over surveillance ignited this week after the German chancellor, Angela Merkel, accused the NSA of monitoring her phone calls, and the subsequent revelation that the agency monitored communications of at least 35 other world leaders.On Friday, Merkel and the French president, François Hollande, agreed to spearhead efforts to make the NSA sign a new code of conduct on how it carried out intelligence operations within the European Union, after EU leaders warned that the international fight against terrorism was being jeopardised by the perception that mass US surveillance was out of control.Fear of diplomatic repercussions were one of the prime reasons given for GCHQ's insistence that its relationships with telecoms firms must be kept private .

Telecoms companies "feared damage to their brands internationally, if the extent of their co-operation with HMG [Her Majesty's government] became apparent", the GCHQ document warned. It added that if intercepts became admissible as evidence in UK courts "many CSPs asserted that they would withdraw their voluntary support".The report stressed that while companies are going beyond what they are required to do under UK law, they are not being asked to violate it.Shami Chakrabarti, Director of Liberty and Anthony Romero Executive Director of the American Civil Liberties Union issued a joint statement stating:"The Guardian's publication of information from Edward Snowden has uncovered a breach of trust by the US and UK Governments on the grandest scale. The newspaper's principled and selective revelations demonstrate our rulers' contempt for personal rights, freedoms and the rule of law.

"Across the globe, these disclosures continue to raise fundamental questions about the lack of effective legal protection against the interception of all our communications."Yet in Britain, that conversation is in danger of being lost beneath self-serving spin and scaremongering, with journalists who dare to question the secret state accused of aiding the enemy."A balance must of course be struck between security and transparency, but that cannot be achieved whilst the intelligence services and their political masters seek to avoid any scrutiny of, or debate about, their actions."The Guardian's decision to expose the extent to which our privacy is being violated should be applauded and not condemned."

Snowden told The Intercept that the latest revelations amounted to unprecedented “smoking-gun attribution for a governmental cyber attack against critical infrastructure.” The Belgacom hack, he said, is the “first documented example to show one EU member state mounting a cyber attack on another…a breathtaking example of the scale of the state-sponsored hacking problem.”

Publicly, Belgacom has played down the extent of the compromise, insisting that only its internal systems were breached and that customers’ data was never found to have been at risk. But secret GCHQ documents show the agency gained access far beyond Belgacom’s internal employee computers and was able to grab encrypted and unencrypted streams of private communications handled by the company. Belgacom invested several million dollars in its efforts to clean-up its systems and beef-up its security after the attack. However, The Intercept has learned that sources familiar with the malware investigation at the company are uncomfortable with how the clean-up operation was handled—and they believe parts of the GCHQ malware were never fully removed.

The revelations about the scope of the hacking operation will likely alarm Belgacom’s customers across the world. The company operates a large number of data links internationally (see interactive map below), and it serves millions of people across Europe as well as officials from top institutions including the European Commission, the European Parliament, and the European Council. The new details will also be closely scrutinized by a federal prosecutor in Belgium, who is currently carrying out a criminal investigation into the attack on the company. Sophia in ’t Veld, a Dutch politician who chaired the European Parliament’s recent inquiry into mass surveillance exposed by Snowden, told The Intercept that she believes the British government should face sanctions if the latest disclosures are proven.

What sets the secret British infiltration of Belgacom apart is that it was perpetrated against a close ally—and is backed up by a series of top-secret documents, which The Intercept is now publishing.

Between 2009 and 2011, GCHQ worked with its allies to develop sophisticated new tools and technologies it could use to scan global networks for weaknesses and then penetrate them. According to top-secret GCHQ documents, the agency wanted to adopt the aggressive new methods in part to counter the use of privacy-protecting encryption—what it described as the “encryption problem.” When communications are sent across networks in encrypted format, it makes it much harder for the spies to intercept and make sense of emails, phone calls, text messages, internet chats, and browsing sessions. For GCHQ, there was a simple solution. The agency decided that, where possible, it would find ways to hack into communication networks to grab traffic before it’s encrypted.

The Snowden documents show that GCHQ wanted to gain access to Belgacom so that it could spy on phones used by surveillance targets travelling in Europe. But the agency also had an ulterior motive. Once it had hacked into Belgacom’s systems, GCHQ planned to break into data links connecting Belgacom and its international partners, monitoring communications transmitted between Europe and the rest of the world. A map in the GCHQ documents, named “Belgacom_connections,” highlights the company’s reach across Europe, the Middle East, and North Africa, illustrating why British spies deemed it of such high value.

Documents published with this article: Automated NOC detection Mobile Networks in My NOC World Making network sense of the encryption problem Stargate CNE requirements NAC review – October to December 2011 GCHQ NAC review – January to March 2011 GCHQ NAC review – April to June 2011 GCHQ NAC review – July to September 2011 GCHQ NAC review – January to March 2012 GCHQ Hopscotch Belgacom connections

Optic Nerve, the documents provided by NSA whistleblower Edward Snowden show, began as a prototype in 2008 and was still active in 2012, according to an internal GCHQ wiki page accessed that year.The system, eerily reminiscent of the telescreens evoked in George Orwell's 1984, was used for experiments in automated facial recognition, to monitor GCHQ's existing targets, and to discover new targets of interest. Such searches could be used to try to find terror suspects or criminals making use of multiple, anonymous user IDs.Rather than collecting webcam chats in their entirety, the program saved one image every five minutes from the users' feeds, partly to comply with human rights legislation, and also to avoid overloading GCHQ's servers. The documents describe these users as "unselected" – intelligence agency parlance for bulk rather than targeted collection.One document even likened the program's "bulk access to Yahoo webcam images/events" to a massive digital police mugbook of previously arrested individuals.

Programs like Optic Nerve, which collect information in bulk from largely anonymous user IDs, are unable to filter out information from UK or US citizens. Unlike the NSA, GCHQ is not required by UK law to "minimize", or remove, domestic citizens' information from its databases. However, additional legal authorisations are required before analysts can search for the data of individuals likely to be in the British Isles at the time of the search.There are no such legal safeguards for searches on people believed to be in the US or the other allied "Five Eyes" nations – Australia, New Zealand and Canada.

The documents also show that GCHQ trialled automatic searches based on facial recognition technology, for people resembling existing GCHQ targets: "[I]f you search for similar IDs to your target, you will be able to request automatic comparison of the face in the similar IDs to those in your target's ID".

Interviews with the UK source and the NSA whistleblower Edward Snowden raise questions about whether the programme:■ Exploits existing law which was passed by parliament without any anticipation that it would be used for this purpose.■ For the first time allows GCHQ to process bulk internal UK traffic which is routed overseas via these cables.■ Allows the NSA to engage in bulk intercepts of internal US traffic which would be forbidden in its own territory.■ Functions with no effective oversight.

There are similar concerns about the role of the NSA. It could have chosen to attach probes to the North American end of the cables and documents shown to the Guardian by Edward Snowden suggest that key elements of the Tempora filtering process were designed by the NSA. Instead, the NSA agency has exported its computer programs and 250 of its analysts to operate the system from the UK.Initial inquiries by the Guardian have failed to explain why this has happened, but US legislators are likely to want to check whether the NSA has sought to bypass legal or policy requirements which restrict its activity in the US. This will be particularly sensitive if it is confirmed that Tempora is also analysing internal US traffic.The UK source challenges the official justification for the programme; that it is necessary for the fight against terrorism and serious crime: "This is not scoring very high against those targets, because they are wise to the monitoring of their communications. If the terrorists are wise to it, why are we increasing the capability?

Defenders insist that the mass of data is heavily filtered by the programme so that only that relating to legitimate targets is analysed.However, there are doubts about the effectiveness of this. First, according to the UK source, "written definitions for targeting and filtering are very elastic. They are wide open to interpretation." The target areas defined by the Ripa certificates are secret.Second, there is further room for interpretation when human analysts become involved in using the filtered intelligence to produce what are known as "contact chains". "Here is target A. But who is A talking to? Now we're into B and C and D." If analysts believe it is proportional, they can look at all the traffic – content and metadata – relating to all of the target's contact." GCHQ audits a sample of its analysts' work – believed to be 5% every six months – but even the statistical results of these audits are also secret.

Beyond the detail of the operation of the programme, there is a larger, long-term anxiety, clearly expressed by the UK source: "If there was the wrong political change, it could be very dangerous. All you need is to have the wrong government in place. It is capable of abuse because there is no independent scrutiny."

In a 32-page opinion, the leading public law barrister Jemima Stratford QC raises a series of concerns about the legality and proportionality of GCHQ's work, and the lack of safeguards for protecting privacy.

The opinion notes that the UK has not adopted the doctrine of "anticipatory self-defence" in the same way as the US to provide legal cover for drone strikes in countries where it is not involved in an international armed conflict."Accordingly, in our view, if GCHQ transferred data to the NSA in the knowledge that it would or might be used for targeting drone strikes, that transfer is probably unlawful," the advice states."The transferor would be an accessory to murder for the purposes of domestic law … We consider that, pursuant to the transfer, the agent is likely to become an accessory to murder."Watson said he would be submitting the legal opinion to the parliamentary intelligence and security committee, which is undertaking an inquiry into mass surveillance."MPs now have strong independent advice questioning the legality of major UK intelligence programmes," he said.

The advice concludes: "In short, the rules concerning communications data are too uncertain and do not provide sufficient clarity to be in accordance with the law … we consider the mass interception of communications via a transatlantic cable to be unlawful, and that these conclusions would apply even if some or all of the interception is taking place outside UK territorial waters."Leaving decisions about whether data can be shared with agencies abroad to the "unfettered discretion" of ministers is also a probable breach of the convention, the advice warns.

"First, the transfer of private data is a significant interference with an individual's article 8 rights. That interference will only be lawful when proportionate."Secondly, the ECHR has held on more than one occasion that surveillance, and the use of surveillance data, is an area in which governments must conduct themselves in a transparent and 'predictable' manner. The current framework is uncertain: it relies on the discretion of one individual."Thirdly, on a pragmatic level,there is a real possibility that the NSA might function as GCHQ's unofficial 'backup' service. If GCHQ is not entitled to hold onto data itself, it might transfer it to the NSA. In time, and if relevant, that data might be transferred back to GCHQ. Without strong guidelines and scrutiny, the two services might support each other to (in effect) circumvent the requirements of their domestic legislation."The opinion adds: "If GCHQ transfers communications data to other governments it does so without any statutory restrictions. Such transfers are a disproportionate interference with the article 8 rights of the individuals concerned. There are no restrictions, checks or restraints on the transfer of that data."

At its most extreme, the advice raises issues about the possible vulnerability of staff at GCHQ if it could be proved that intelligence used for US drone strikes against "non-combatants" had been passed on or supplied by the British before being used in a missile attack."An individual involved in passing that information is likely to be an accessory to murder. It is well arguable, on a variety of different bases, that the government is obliged to take reasonable steps to investigate that possibility," the advice says.

"If ministers are prepared to allow GCHQ staff to be potential accessories to murder, they must be very clear that they are responsible for allowing it. We have seen a step change in mass covert surveillance and intelligence gathering, underpinned on dubious legal grounds and with virtually no parliamentary oversight. "The leadership of all the main parties should stop turning a blind eye to a programme that has far-reaching consequences around the globe."

The decision was a disappointing one, and we’ll soon appeal to the European court of human rights. But it left us with a small glimmer of hope. The tribunal said that it was lawful for GCHQ and the NSA to swap and share surveillance material only because GCHQ has secret internal policies that it reluctantly disclosed in response to Privacy International’s case. Now that those secret policies are no longer secret, the court reasoned, the British public know what’s going on, and that in itself must make those activities lawful. It must follow, therefore, that before those policies were public – prior to Edward Snowden’s disclosures, and our case in the IPT – GCHQ was acting outside the law. Complicated reasoning aside, this finding was a genuine – and rare – success. The tribunal agreed, and we today have a firm statement that the intelligence services were acting completely out of bounds. It is not the judgment we would have liked – that we still hope to get from the European court of human rights in Strasbourg later this year – but it is a significant victory against an arm of the state that has rarely been forced to account for its wrongdoings.

It is a vindication of Snowden, and all those who put their careers – and even their lives – on the line to ensure the truth was told. It is a huge encouragement to civil society organisations like Privacy International, which often spend years locked in David and Goliath battles, depleting their funds and their morale to perform the essential role of holding truth to power. In years to come we will look back on today as an essential victory against the surveillance state. Here at Privacy International, we humbly hope that perhaps we will also look back at this day as a turning of the tide; the day when the seemingly uncontrollable advancement of state intrusion into individuals’ lives was halted, and when internet users reclaimed some of the power in their fight for privacy, security and free expression.

The NSA, like GCHQ, has studied Kaspersky Lab’s software for weaknesses. In 2008, an NSA research team discovered that Kaspersky software was transmitting sensitive user information back to the company’s servers, which could easily be intercepted and employed to track users, according to a draft of a top-secret report. The information was embedded in “User-Agent” strings included in the headers of Hypertext Transfer Protocol, or HTTP, requests. Such headers are typically sent at the beginning of a web request to identify the type of software and computer issuing the request.

According to the draft report, NSA researchers found that the strings could be used to uniquely identify the computing devices belonging to Kaspersky customers. They determined that “Kaspersky User-Agent strings contain encoded versions of the Kaspersky serial numbers and that part of the User-Agent string can be used as a machine identifier.” They also noted that the “User-Agent” strings may contain “information about services contracted for or configurations.” Such data could be used to passively track a computer to determine if a target is running Kaspersky software and thus potentially susceptible to a particular attack without risking detection.

Another way the NSA targets foreign anti-virus companies appears to be to monitor their email traffic for reports of new vulnerabilities and malware. A 2010 presentation on “Project CAMBERDADA” shows the content of an email flagging a malware file, which was sent to various anti-virus companies by François Picard of the Montréal-based consulting and web hosting company NewRoma. The presentation of the email suggests that the NSA is reading such messages to discover new flaws in anti-virus software. Picard, contacted by The Intercept, was unaware his email had fallen into the hands of the NSA. He said that he regularly sends out notification of new viruses and malware to anti-virus companies, and that he likely sent the email in question to at least two dozen such outfits. He also said he never sends such notifications to government agencies. “It is strange the NSA would show an email like mine in a presentation,” he added.

The NSA presentation goes on to state that its signals intelligence yields about 10 new “potentially malicious files per day for malware triage.” This is a tiny fraction of the hostile software that is processed. Kaspersky says it detects 325,000 new malicious files every day, and an internal GCHQ document indicates that its own system “collect[s] around 100,000,000 malware events per day.” After obtaining the files, the NSA analysts “[c]heck Kaspersky AV to see if they continue to let any of these virus files through their Anti-Virus product.” The NSA’s Tailored Access Operations unit “can repurpose the malware,” presumably before the anti-virus software has been updated to defend against the threat.

The Project CAMBERDADA presentation lists 23 additional AV companies from all over the world under “More Targets!” Those companies include Check Point software, a pioneering maker of corporate firewalls based Israel, whose government is a U.S. ally. Notably omitted are the American anti-virus brands McAfee and Symantec and the British company Sophos.

As government spies have sought to evade anti-virus software, the anti-virus firms themselves have exposed malware created by government spies. Among them, Kaspersky appears to be the sharpest thorn in the side of government hackers. In the past few years, the company has proven to be a prolific hunter of state-sponsored malware, playing a role in the discovery and/or analysis of various pieces of malware reportedly linked to government hackers, including the superviruses Flame, which Kaspersky flagged in 2012; Gauss, also detected in 2012; Stuxnet, discovered by another company in 2010; and Regin, revealed by Symantec. In February, the Russian firm announced its biggest find yet: the “Equation Group,” an organization that has deployed espionage tools widely believed to have been created by the NSA and hidden on hard drives from leading brands, according to Kaspersky. In a report, the company called it “the most advanced threat actor we have seen” and “probably one of the most sophisticated cyber attack groups in the world.”

Hacks deployed by the Equation Group operated undetected for as long as 14 to 19 years, burrowing into the hard drive firmware of sensitive computer systems around the world, according to Kaspersky. Governments, militaries, technology companies, nuclear research centers, media outlets and financial institutions in 30 countries were among those reportedly infected. Kaspersky estimates that the Equation Group could have implants in tens of thousands of computers, but documents published last year by The Intercept suggest the NSA was scaling up their implant capabilities to potentially infect millions of computers with malware. Kaspersky’s adversarial relationship with Western intelligence services is sometimes framed in more sinister terms; the firm has been accused of working too closely with the Russian intelligence service FSB. That accusation is partly due to the company’s apparent success in uncovering NSA malware, and partly due to the fact that its founder, Eugene Kaspersky, was educated by a KGB-backed school in the 1980s before working for the Russian military.

Kaspersky has repeatedly denied the insinuations and accusations. In a recent blog post, responding to a Bloomberg article, he complained that his company was being subjected to “sensationalist … conspiracy theories,” sarcastically noting that “for some reason they forgot our reports” on an array of malware that trace back to Russian developers. He continued, “It’s very hard for a company with Russian roots to become successful in the U.S., European and other markets. Nobody trusts us — by default.”

Documents published with this article: Kaspersky User-Agent Strings — NSA Project CAMBERDADA — NSA NDIST — GCHQ’s Developing Cyber Defence Mission GCHQ Application for Renewal of Warrant GPW/1160 Software Reverse Engineering — GCHQ Reverse Engineering — GCHQ Wiki Malware Analysis & Reverse Engineering — ACNO Skill Levels — GCHQ

Among the programs said to have been operating were Turbine, which automates the injection of data and can infect millions of machines and Warrior Pride, which enables microphones on iPhones and Android devices to be remotely activated.

GCHQ vacuums up the website browsing histories using “probes” that tap into the international fiber-optic cables that transport Internet traffic across the world. A huge volume of the Internet data GCHQ collects flows directly into a massive repository named Black Hole, which is at the core of the agency’s online spying operations, storing raw logs of intercepted material before it has been subject to analysis. Black Hole contains data collected by GCHQ as part of bulk “unselected” surveillance, meaning it is not focused on particular “selected” targets and instead includes troves of data indiscriminately swept up about ordinary people’s online activities. Between August 2007 and March 2009, GCHQ documents say that Black Hole was used to store more than 1.1 trillion “events”  a term the agency uses to refer to metadata records  with about 10 billion new entries added every day. As of March 2009, the largest slice of data Black Hole held  41 percent  was about people’s Internet browsing histories. The rest included a combination of email and instant messenger records, details about search engine queries, information about social media activity, logs related to hacking operations, and data on people’s use of tools to browse the Internet anonymously.

Throughout this period, as smartphone sales started to boom, the frequency of people’s Internet use was steadily increasing. In tandem, British spies were working frantically to bolster their spying capabilities, with plans afoot to expand the size of Black Hole and other repositories to handle an avalanche of new data. By 2010, according to the documents, GCHQ was logging 30 billion metadata records per day. By 2012, collection had increased to 50 billion per day, and work was underway to double capacity to 100 billion. The agency was developing “unprecedented” techniques to perform what it called “population-scale” data mining, monitoring all communications across entire countries in an effort to detect patterns or behaviors deemed suspicious. It was creating what it saidwould be, by 2013, “the world’s biggest” surveillance engine “to run cyber operations and to access better, more valued data for customers to make a real world difference.” HERE WAS A SIMPLE AIM at the heart of the top-secret program: Record the website browsing habits of “every visible user on the Internet.” Before long, billions of digital records about ordinary people’s online activities were being stored every day. Among them were details cataloging visits to porn, social media and news websites, search engines, chat forums, and blogs.

The mass surveillance operation — code-named KARMA POLICE — was launched by British spies about seven years ago without any public debate or scrutiny. It was just one part of a giant global Internet spying apparatus built by the United Kingdom’s electronic eavesdropping agency, Government Communications Headquarters, or GCHQ. The revelations about the scope of the British agency’s surveillance are contained in documents obtained by The Intercept from National Security Agency whistleblower Edward Snowden. Previous reports based on the leaked files have exposed how GCHQ taps into Internet cables to monitor communications on a vast scale, but many details about what happens to the data after it has been vacuumed up have remained unclear.

(The GCHQ document also describes a practice called “credential harvesting,” which NBC described as an effort to “select journalists who could be used to spread information” that the government wants distributed. According to the NBC report, GCHQ agents would employ “electronic snooping to identify non-British journalists who would then be manipulated to feed information to the target of a covert campaign.” Then, “the journalist’s job would provide access to the targeted individual, perhaps for an interview.” Anonymous sources that NBC didn’t characterize claimed at the time that GCHQ had not employed the technique.) Whether governments should be in the business of publicly disseminating political propaganda at all is itself a controversial question. Such activities are restricted by law in many countries, including the U.S. In 2008, The New York Times’ David Barstow won a Pulitzer Prize for exposing a domestic effort coordinated by the Pentagon whereby retired U.S. generals posed as “independent analysts” employed by American television networks and cable news outlets as they secretly coordinated their messaging with the Pentagon.

For instance: One of the programs described by the newly released GCHQ document is dubbed “Royal Concierge,” under which the British agency intercepts email confirmations of hotel reservations to enable it to subject hotel guests to electronic monitoring. It also contemplates how to “influence the hotel choice” of travelers and to determine whether they stay at “SIGINT friendly” hotels. The document asks: “Can we influence the hotel choice? Can we cancel their visit?” Previously, der Spiegel and NBC News both independently confirmed that the “Royal Concierge” program has been implemented and extensively used. The German magazine reported that “for more than three years, GCHQ has had a system to automatically monitor hotel bookings of at least 350 upscale hotels around the world in order to target, search, and analyze reservations to detect diplomats and government officials.” NBC reported that “the intelligence agency uses the information to spy on human targets through ‘close access technical operations,’ which can include listening in on telephone calls and tapping hotel computers as well as sending intelligence officers to observe the targets in person at the hotels.”

The GCHQ document we are publishing today expressly contemplates exploiting social media venues such as Twitter, as well as other communications venues including email, to seed state propaganda–GHCQ’s word, not mine–across the internet:

Because American law bars the government from employing political propaganda domestically, that program was likely illegal, though no legal accountability was ever brought to bear (despite all sorts of calls for formal investigations). Barack Obama, a presidential candidate at the time, pronounced himself in a campaign press release “deeply disturbed” by the Pentagon program, which he said “sought to manipulate the public’s trust.” Propagandizing foreign populations has generally been more legally acceptable. But it is difficult to see how government propaganda can be segregated from domestic consumption in the digital age. If American intelligence agencies are adopting the GCHQ’s tactics of “crafting messaging campaigns to go ‘viral’,” the legal issue is clear: A “viral” online propaganda campaign, by definition, is almost certain to influence its own citizens as well as those of other countries.

But these documents, along with the AP’s exposure of the sham “Cuban Twitter” program, underscore how aggressively western governments are seeking to exploit the internet as a means to manipulate political activity and shape political discourse. Those programs, carried out in secrecy and with little accountability (it seems nobody in Congress knew of the “Cuban Twitter” program in any detail) threaten the integrity of the internet itself, as state-disseminated propaganda masquerades as free online speech and organizing. There is thus little or no ability for an internet user to know when they are being covertly propagandized by their government, which is precisely what makes it so appealing to intelligence agencies, so powerful, and so dangerous.

According to the reports, Cable & Wireless also appears to have helped GCHQ obtain data from a rival foreign communications company, India’s Reliance Communications, enabling the spies to sweep up communications sent by millions of internet users worldwide through a Reliance-owned cable that stretches from England across Asia and the Middle East. This so-called “access point” for GCHQ was named Nigella and located near an agency surveillance base in Bude, Cornwall (pictured above). Reliance did not respond to a request for comment. In July 2012, the multinational phone company Vodafone bought Cable & Wireless for about $1.5 billion. The documents indicate that the Nigella surveillance access point remained active as of April 2013. Vodafone said in a statement that it complies with the law and does not give “direct access” to its cables. The company says it is compelled to provide certain access to data based on warrants issued by the government.

One slide from a May 2010 NSA presentation on getting data from smartphones – breathlessly titled "Golden Nugget!" – sets out the agency's "perfect scenario": "Target uploading photo to a social media site taken with a mobile device. What can we get?"The question is answered in the notes to the slide: from that event alone, the agency said it could obtain a "possible image", email selector, phone, buddy lists, and "a host of other social working data as well as location".

The latest disclosures could also add to mounting public concern about how the technology sector collects and uses information, especially for those outside the US, who enjoy fewer privacy protections than Americans. A January poll for the Washington Post showed 69% of US adults were already concerned about how tech companies such as Google used and stored their information.The documents do not make it clear how much of the information that can be taken from apps is routinely collected, stored or searched, nor how many users may be affected. The NSA says it does not target Americans and its capabilities are deployed only against "valid foreign intelligence targets".The documents do set out in great detail exactly how much information can be collected from widely popular apps. One document held on GCHQ's internal Wikipedia-style guide for staff details what can be collected from different apps. Though it uses Android apps for most of its examples, it suggests much of the same data could be taken from equivalent apps on iPhone or other platforms.The GCHQ documents set out examples of what information can be extracted from different ad platforms, using perhaps the most popular mobile phone game of all time, Angry Birds – which has reportedly been downloaded more than 1.7bn times – as a case study.

Other apps choose to transmit much more data, meaning the agency could potentially net far more. One mobile ad platform, Millennial Media, appeared to offer particularly rich information. Millennial Media's website states it has partnered with Rovio on a special edition of Angry Birds; with Farmville maker Zynga; with Call of Duty developer Activision, and many other major franchises.

An unclassified table accompanying the report states that encryption is the “[b]est defense to protect data”, especially if made particularly strong through “multi-factor authentication” – similar to two-step verification used by Google and others for email – or biometrics. These measures remain all but impossible to crack, even for GCHQ and the NSA. The report warned: “Almost all current and potential adversaries – nations, criminal groups, terrorists, and individual hackers – now have the capability to exploit, and in some cases attack, unclassified access-controlled US and allied information systems.” It further noted that the “scale of detected compromises indicates organisations should assume that any controlled but unclassified networks of intelligence, operational or commercial value directly accessible from the internet are already potentially compromised by foreign adversaries”.

The report had some cause for optimism, especially in the light of Google and other US tech giants having in the months prior greatly increased their use of encryption efforts. “We assess with high confidence that security best practices applied to target networks would prevent the vast majority of intrusions,” it concluded. Official UK government security advice still recommends encryption among a range of other tools for effective network and information defence. However, end-to-end encryption – which means only the two people communicating with each other, and not the company carrying the message, can decode it – is problematic for intelligence agencies as it makes even warranted collection much more difficult.

The previous week, a day after the attack on the Charlie Hebdo office in Paris, the MI5 chief, Andrew Parker, called for new powers and warned that new technologies were making it harder to track extremists. In November, the head of GCHQ, Robert Hannigan, said US social media giants had become the “networks of choice” for terrorists. Chris Soghoian, principal senior policy analyst at the American Civil Liberties Union, said attempts by the British government to force US companies to weaken encryption faced many hurdles.

The Guardian, New York Times and ProPublica have previously reported the intelligence agencies’ broad efforts to undermine encryption and exploit rather than reveal vulnerabilities. This prompted Obama’s NSA review panel to warn that the agency’s conflicting missions caused problems, and so recommend that its cyber-security responsibilities be removed to prevent future issues.

The memo requested a renewal of the legal warrant allowing GCHQ to “modify” commercial software in violation of licensing agreements. The document cites examples of software the agency had hacked, including commonly used software to run web forums, and website administration tools. Such software are widely used by companies and individuals around the world. The document also said the agency had developed “capability against Cisco routers”, which would “allow us to re-route selected traffic across international links towards GCHQ’s passive collection systems”. GCHQ had also been working to “exploit” the anti-virus software Kaspersky, the document said. The report contained no information on the nature of the vulnerabilities found by the agency.

Michael Beckerman, president and CEO of the Internet Association, a lobby group that represents Facebook, Google, Reddit, Twitter, Yahoo and other tech companies, said: “Just as governments have a duty to protect to the public from threats, internet services have a duty to our users to ensure the security and privacy of their data. That’s why internet services have been increasing encryption security.”

A cable called ESAT 1, which goes from Kilmore Quay in Wexford and lands at Sennen Cove in Cornwall, is on the list, as is ESAT2, another cable that runs from Sandymount in Dublin to Southport, north of Liverpool in the UK. The German newspaper Süddeutsche Zeitung released the documents as part of its report on a new cache of documents from Mr Snowden. It focused on how telecom firm Cable & Wireless, which was acquired by Vodafone in 2012, “played a key role in establishing one of the Government Communications Headquarters’ (GCHQ) most controversial surveillance programmes”.

The newspaper said that according to the documents, Cable & Wireless “actively shaped and provided the most data to GCHQ mass surveillance programmes, and received millions of pounds in compensation”.Vodafone says it does not go beyond what is required under the law when responding to demands from any agency for access to customer data. Telecommunications companies can be legally compelled to co-operate with intelligence agencies in providing access to cables and also forbidden to disclose their involvement.