Group items tagged

Then we put the data up, but the problem with Solr was it didn’t have a user interface, so we used Project Blacklight, which is open source software normally used by librarians. We used it for the journalists. It’s simple because it allows you to do faceted search—so, for example, you can facet by the folder structure of the leak, by years, by type of file. There were more complex things—it supports queries in regular expressions, so the more advanced users were able to search for documents with a certain pattern of numbers that, for example, passports use. You could also preview and download the documents. ICIJ open-sourced the code of our document processing chain, created by our web developer Matthew Caruana Galizia. We also developed a batch-searching feature. So say you were looking for politicians in your country—you just run it through the system, and you upload your list to Blacklight and you would get a CSV back saying yes, there are matches for these names—not only exact matches, but also matches based on proximity. So you would say “I want Mar Cabra proximity 2” and that would give you “Mar Cabra,” “Mar whatever Cabra,” “Cabra, Mar,”—so that was good, because very quickly journalists were able to see… I have this list of politicians and they are in the data!

Last Sunday, April 3, the first stories emerging from the leaked dataset known as the Panama Papers were published by a global partnership of news organizations working in coordination with the International Consortium of Investigative Journalists, or ICIJ. As we begin the second week of reporting on the leak, Iceland’s Prime Minister has been forced to resign, Germany has announced plans to end anonymous corporate ownership, governments around the world launched investigations into wealthy citizens’ participation in tax havens, the Russian government announced that the investigation was an anti-Putin propaganda operation, and the Chinese government banned mentions of the leak in Chinese media. As the ICIJ-led consortium prepares for its second major wave of reporting on the Panama Papers, we spoke with Mar Cabra, editor of ICIJ’s Data & Research unit and lead coordinator of the data analysis and infrastructure work behind the leak. In our conversation, Cabra reveals ICIJ’s years-long effort to build a series of secure communication and analysis platforms in support of genuinely global investigative reporting collaborations.

For communication, we have the Global I-Hub, which is a platform based on open source software called Oxwall. Oxwall is a social network, like Facebook, which has a wall when you log in with the latest in your network—it has forum topics, links, you can share files, and you can chat with people in real time.

When AT&T and Time Warner announced their $85.4 billion deal in October, lawmakers greeted the acquisition frostily. Now their tone is changing.At a hearing on Capitol Hill on Wednesday that was being closely watched for how mega-mergers will be viewed in the coming Trump administration, members of a Senate Judiciary subcommittee that oversees regulatory agencies that decide on mergers said the deal merited tough scrutiny. The chief executives of AT&T and Time Warner were grilled at the hearing about a range of issues related to the deal.But in a change from previous comments, lawmakers also questioned whether traditional ways of evaluating mergers are growing outdated as Silicon Valley companies like Facebook and Google become massive media platforms that threaten the television industry. Their tone was more circumspect than those that immediately followed the deal’s announcement, when lawmakers had been more critical.

The Senate Judiciary Committee held an open hearing today on the FISA Amendments Act, the law that ostensibly authorizes the digital surveillance of hundreds of millions of people both in the United States and around the world. Section 702 of the law, scheduled to expire next year, is designed to allow U.S. intelligence services to collect signals intelligence on foreign targets related to our national security interests. However—thanks to the leaks of many whistleblowers including Edward Snowden, the work of investigative journalists, and statements by public officials—we now know that the FISA Amendments Act has been used to sweep up data on hundreds of millions of people who have no connection to a terrorist investigation, including countless Americans. What do we mean by “countless”? As became increasingly clear in the hearing today, the exact number of Americans impacted by this surveillance is unknown. Senator Franken asked the panel of witnesses, “Is it possible for the government to provide an exact count of how many United States persons have been swept up in Section 702 surveillance? And if not the exact count, then what about an estimate?”

The lack of information makes rigorous oversight of the programs all but impossible. As Senator Franken put it in the hearing today, “When the public lacks even a rough sense of the scope of the government’s surveillance program, they have no way of knowing if the government is striking the right balance, whether we are safeguarding our national security without trampling on our citizens’ fundamental privacy rights. But the public can’t know if we succeed in striking that balance if they don’t even have the most basic information about our major surveillance programs."  Senator Patrick Leahy also questioned the panel about the “minimization procedures” associated with this type of surveillance, the privacy safeguard that is intended to ensure that irrelevant data and data on American citizens is swiftly deleted. Senator Leahy asked the panel: “Do you believe the current minimization procedures ensure that data about innocent Americans is deleted? Is that enough?”  David Medine, who recently announced his pending retirement from the Privacy and Civil Liberties Oversight Board, answered unequivocally:

Elizabeth Goitein, the Brennan Center director whose articulate and thought-provoking testimony was the highlight of the hearing, noted that at this time an exact number would be difficult to provide. However, she asserted that an estimate should be possible for most if not all of the government’s surveillance programs. None of the other panel participants—which included David Medine and Rachel Brand of the Privacy and Civil Liberties Oversight Board as well as Matthew Olsen of IronNet Cybersecurity and attorney Kenneth Wainstein—offered an estimate. Today’s hearing reaffirmed that it is not only the American people who are left in the dark about how many people or accounts are impacted by the NSA’s dragnet surveillance of the Internet. Even vital oversight committees in Congress like the Senate Judiciary Committee are left to speculate about just how far-reaching this surveillance is. It's part of the reason why we urged the House Judiciary Committee to demand that the Intelligence Community provide the public with a number. 

The Common Crawl Foundation’s repository of openly and freely accessible web crawl data is about to go live as a Public Data Set on Amazon Web Services.

Elbaz’ goal in developing the repository: “You can’t access, let alone download, the Google or the Bing crawl data. So certainly we’re differentiated in being very open and transparent about what we’re crawling and actually making it available to developers,” he says. “You might ask why is it going to be revolutionary to allow many more engineers and researchers and developers and students access to this data, whereas historically you have to work for one of the big search engines…. The question is, the world has the largest-ever corpus of knowledge out there on the web, and is there more that one can do with it than Google and Microsoft and a handful of other search engines are already doing? And the answer is unquestionably yes. ”

Common Crawl’s data already is stored on Amazon’s S3 service, but now Amazon will be providing the storage space for free through the Public Data Set program. Not only does that remove from Common Crawl the storage burden and costs for hosting its crawl of 5 billion web pages – some 50 or 60 terabytes large – but it should make it easier for users to access the data, and remove the bandwidth-related costs they might incur for downloads. Users won’t have to deal with setting up accounts, being responsible for bandwidth bills incurred, and more complex authentication processes.

The two organizations currently responsible for the development of HTML have decided on a degree of separation and this means that in the future there will be two versions of HTML5 - the snapshot and the living standard.

In a post to the WHATWG list, the editor of the WHATWG specifications explains: More recently, the goals of the W3C and the WHATWG on the HTML front have diverged a bit as well. The WHATWG effort is focused on developing the canonical description of HTML and related technologies, meaning fixing bugs as we find them adding new features as they become necessary and viable, and generally tracking implementations. The W3C effort, meanwhile, is now focused on creating a snapshot developed according to the venerable W3C process. This led to the chairs of the W3C HTML working group and myself deciding to split the work into two, with a different person responsible for editing the W3C HTML5, canvas, and microdata specifications than is editing the WHATWG specification.

If you think that these two organizations are now going their separate ways and that this means that there will be two HTML5 standards, I think you are likely to be correct.

Google is considering deploying Wi-Fi networks in towns and cities covered by its Google Fiber high-speed Internet service. The disclosure is made in a document Google is circulating to 34 cities that are the next candidates to receive Google Fiber in 2015.

Google Fiber is already available in Provo, Utah, and Kansas City, and is promised soon in Austin, Texas. It delivers a "basic speed" service for no charge, a gigabit-per-second service for US$70 per month and a $120 package that includes a bundle of more than 200 TV channels. Installation costs between nothing and $300. Google has sent the 34 cities that are next in line for Google Fiber a detailed request for information and they have until May 1 to reply.

Specific details of the Wi-Fi plan are not included in the document, which was seen by IDG News Service, but Google says it will be "discussing our Wi-Fi plans and related requirements with your city as we move forward with your city during this planning process."

WASHINGTON (AP) -- The Obama administration has been quietly advising local police not to disclose details about surveillance technology they are using to sweep up basic cellphone data from entire neighborhoods, The Associated Press has learned. Citing security reasons, the U.S. has intervened in routine state public records cases and criminal trials regarding use of the technology. This has resulted in police departments withholding materials or heavily censoring documents in rare instances when they disclose any about the purchase and use of such powerful surveillance equipment. Federal involvement in local open records proceedings is unusual. It comes at a time when President Barack Obama has said he welcomes a debate on government surveillance and called for more transparency about spying in the wake of disclosures about classified federal surveillance programs.

One well-known type of this surveillance equipment is known as a Stingray, an innovative way for law enforcement to track cellphones used by suspects and gather evidence. The equipment tricks cellphones into identifying some of their owners' account information, like a unique subscriber number, and transmitting data to police as if it were a phone company's tower. That allows police to obtain cellphone information without having to ask for help from service providers, such as Verizon or AT&T, and can locate a phone without the user even making a call or sending a text message. But without more details about how the technology works and under what circumstances it's used, it's unclear whether the technology might violate a person's constitutional rights or whether it's a good investment of taxpayer dollars. Interviews, court records and public-records requests show the Obama administration is asking agencies to withhold common information about the equipment, such as how the technology is used and how to turn it on. That pushback has come in the form of FBI affidavits and consultation in local criminal cases.

"These extreme secrecy efforts are in relation to very controversial, local government surveillance practices using highly invasive technology," said Nathan Freed Wessler, a staff attorney with the American Civil Liberties Union, which has fought for the release of these types of records. "If public participation means anything, people should have the facts about what the government is doing to them." Harris Corp., a key manufacturer of this equipment, built a secrecy element into its authorization agreement with the Federal Communications Commission in 2011. That authorization has an unusual requirement: that local law enforcement "coordinate with the FBI the acquisition and use of the equipment." Companies like Harris need FCC authorization in order to sell wireless equipment that could interfere with radio frequencies. A spokesman from Harris Corp. said the company will not discuss its products for the Defense Department and law enforcement agencies, although public filings showed government sales of communications systems such as the Stingray accounted for nearly one-third of its $5 billion in revenue. "As a government contractor, our solutions are regulated and their use is restricted," spokesman Jim Burke said.

The change in the law wrought by the Federal Circuit can also be viewed substantively through the controversy over software patents. Throughout the 1960s, the USPTO refused to award patents for software innovations. However, several of the USPTO’s decisions were overruled by the patent-friendly U.S. Court of Customs and Patent Appeals, which ordered that software patents be granted. In Gottschalk v. Benson (1972) and Parker v. Flook (1978), the U.S. Supreme Court reversed the Court of Customs and Patent Appeals, holding that mathematical algorithms (and therefore software) were not patentable subject matter. In 1981, in Diamond v. Diehr, the Supreme Court upheld a software patent on the grounds that the patent in question involved a physical process—the patent was issued for software used in the molding of rubber. While affirming their prior ruling that mathematical formulas are not patentable in the abstract, the Court held that an otherwise patentable invention did not become unpatentable simply because it utilized a computer.

After only one hour of floor debate, and no allowed amendments, the House of Representatives today passed legislation that opponents believe may give brand new authorization to the U.S. government to conduct domestic dragnets. The USA Freedom Act was approved in a 338-88 vote, with approximately equal numbers of Democrats and Republicans voting against. The bill’s supporters say it will disallow bulk collection of domestic telephone metadata, in which the Foreign Intelligence Surveillance Court has regularly ordered phone companies to turn over such data. The Obama administration claims such collection is authorized by Section 215 of the USA Patriot Act, which is set to expire June 1. However, the U.S. Court of Appeals for the Second Circuit recently held that Section 215 does not provide such authorization. Today’s legislation would prevent the government from issuing such orders for bulk collection and instead rely on telephone companies to store all their metadata — some of which the government could then demand using a “specific selection term” related to foreign terrorism. Bill supporters maintain this would prevent indiscriminate collection.

However, the legislation may not end bulk surveillance and in fact could codify the ability of the government to conduct dragnet data collection. “We’re taking something that was not permitted under regular section 215 … and now we’re creating a whole apparatus to provide for it,” Rep. Justin Amash, R-Mich., said on Tuesday night during a House Rules Committee proceeding. “The language does limit the amount of bulk collection, it doesn’t end bulk collection,” Rep. Amash said, arguing that the problematic “specific selection term” allows for “very large data collection, potentially in the hundreds of thousands of people, maybe even millions.” In a statement posted to Facebook ahead of the vote, Rep. Amash said the legislation “falls woefully short of reining in the mass collection of Americans’ data, and it takes us a step in the wrong direction by specifically authorizing such collection in violation of the Fourth Amendment to the Constitution.”

“While I appreciate a number of the reforms in the bill and understand the need for secure counter-espionage and terrorism investigations, I believe our nation is better served by allowing Section 215 to expire completely and replacing it with a measure that finds a better balance between national security interests and protecting the civil liberties of Americans,” Congressman Ted Lieu, D-Calif., said in a statement explaining his vote against the bill.

today we’re excited to contribute back to the open source community by launching First Look Code, the home for our own open source projects related to privacy, security, data, and journalism. To begin with, First Look Code is the new home for document sanitization software PDF Redact Tools, and we’ve launched a brand new anti-gag order project called AutoCanary.

AutoCanary A warrant canary is a regularly published statement that a company hasn’t received any legal orders that it’s not allowed to talk about, such as a national security letter. Canaries can help prevent web publishers from misleading visitors and prevent tech companies from misleading users when they share data with the government and are prevented from talking about it. One such situation arose — without a canary in place — in 2013, when the U.S. government sent Lavabit, a provider of encrypted email services apparently used by Snowden, a legal request to access Snowden’s email, thwarting some of the very privacy protections Lavabit had promised users. This request included a gag order, so the company was legally prohibited from talking about it. Rather than becoming “complicit in crimes against the American people,” in his words, Lavabit founder Ladar Levison, chose to shut down the service.

Warrant canaries are designed to help companies in this kind of situation. You can see a list of companies that publish warrant canary statements at Canary Watch. As of today, First Look Media is among the companies that publish canaries. We’re happy to announce the first version of AutoCanary, a desktop program for Windows, Mac OS X, and Linux that makes the process of generating machine-readable, digitally-signed warrant canary statements simpler. Read more about AutoCanary on its new website.

One system builds profiles showing people’s web browsing histories. Another analyzes instant messenger communications, emails, Skype calls, text messages, cell phone locations, and social media interactions. Separate programs were built to keep tabs on “suspicious” Google searches and usage of Google Maps. The surveillance is underpinned by an opaque legal regime that has authorized GCHQ to sift through huge archives of metadata about the private phone calls, emails and Internet browsing logs of Brits, Americans, and any other citizens  all without a court order or judicial warrant.

The power of KARMA POLICE was illustrated in 2009, when GCHQ launched a top-secret operation to collect intelligence about people using the Internet to listen to radio shows. The agency used a sample of nearly 7 million metadata records, gathered over a period of three months, to observe the listening habits of more than 200,000 people across 185 countries, including the U.S., the U.K., Ireland, Canada, Mexico, Spain, the Netherlands, France, and Germany.

GCHQ’s documents indicate that the plans for KARMA POLICE were drawn up between 2007 and 2008. The system was designed to provide the agency with “either (a) a web browsing profile for every visible user on the Internet, or (b) a user profile for every visible website on the Internet.” The origin of the surveillance system’s name is not discussed in the documents. But KARMA POLICE is also the name of a popular song released in 1997 by the Grammy Award-winning British band Radiohead, suggesting the spies may have been fans. A verse repeated throughout the hit song includes the lyric, “This is what you’ll get, when you mess with us.”

The Obama administration fought a legal battle against Google to secretly obtain the email records of a security researcher and journalist associated with WikiLeaks. Newly unsealed court documents obtained by The Intercept reveal the Justice Department won an order forcing Google to turn over more than one year’s worth of data from the Gmail account of Jacob Appelbaum (pictured above), a developer for the Tor online anonymity project who has worked with WikiLeaks as a volunteer. The order also gagged Google, preventing it from notifying Appelbaum that his records had been provided to the government. The surveillance of Appelbaum’s Gmail account was tied to the Justice Department’s long-running criminal investigation of WikiLeaks, which began in 2010 following the transparency group’s publication of a large cache of U.S. government diplomatic cables. According to the unsealed documents, the Justice Department first sought details from Google about a Gmail account operated by Appelbaum in January 2011, triggering a three-month dispute between the government and the tech giant. Government investigators demanded metadata records from the account showing email addresses of those with whom Appelbaum had corresponded between the period of November 2009 and early 2011; they also wanted to obtain information showing the unique IP addresses of the computers he had used to log in to the account.

The Justice Department argued in the case that Appelbaum had “no reasonable expectation of privacy” over his email records under the Fourth Amendment, which protects against unreasonable searches and seizures. Rather than seeking a search warrant that would require it to show probable cause that he had committed a crime, the government instead sought and received an order to obtain the data under a lesser standard, requiring only “reasonable grounds” to believe that the records were “relevant and material” to an ongoing criminal investigation. Google repeatedly attempted to challenge the demand, and wanted to immediately notify Appelbaum that his records were being sought so he could have an opportunity to launch his own legal defense. Attorneys for the tech giant argued in a series of court filings that the government’s case raised “serious First Amendment concerns.” They noted that Appelbaum’s records “may implicate journalistic and academic freedom” because they could “reveal confidential sources or information about WikiLeaks’ purported journalistic or academic activities.” However, the Justice Department asserted that “journalists have no special privilege to resist compelled disclosure of their records, absent evidence that the government is acting in bad faith,” and refused to concede Appelbaum was in fact a journalist. It claimed it had acted in “good faith throughout this criminal investigation, and there is no evidence that either the investigation or the order is intended to harass the … subscriber or anyone else.” Google’s attempts to fight the surveillance gag order angered the government, with the Justice Department stating that the company’s “resistance to providing the records” had “frustrated the government’s ability to efficiently conduct a lawful criminal investigation.”

Google accused the government of hyperbole and argued that the backlash over the Twitter order did not justify secrecy related to the Gmail surveillance. “Rather than demonstrating how unsealing the order will harm its well-publicized investigation, the government lists a parade of horribles that have allegedly occurred since it unsealed the Twitter order, yet fails to establish how any of these developments could be further exacerbated by unsealing this order,” wrote Google’s attorneys. “The proverbial toothpaste is out of the tube, and continuing to seal a materially identical order will not change it.” But Google’s attempt to overturn the gag order was denied by magistrate judge Ivan D. Davis in February 2011. The company launched an appeal against that decision, but this too was rebuffed, in March 2011, by District Court judge Thomas Selby Ellis, III.

The National Security Agency and its British counterpart, Government Communications Headquarters, have worked to subvert anti-virus and other security software in order to track users and infiltrate networks, according to documents from NSA whistleblower Edward Snowden. The spy agencies have reverse engineered software products, sometimes under questionable legal authority, and monitored web and email traffic in order to discreetly thwart anti-virus software and obtain intelligence from companies about security software and users of such software. One security software maker repeatedly singled out in the documents is Moscow-based Kaspersky Lab, which has a holding registered in the U.K., claims more than 270,000 corporate clients, and says it protects more than 400 million people with its products. British spies aimed to thwart Kaspersky software in part through a technique known as software reverse engineering, or SRE, according to a top-secret warrant renewal request. The NSA has also studied Kaspersky Lab’s software for weaknesses, obtaining sensitive customer information by monitoring communications between the software and Kaspersky servers, according to a draft top-secret report. The U.S. spy agency also appears to have examined emails inbound to security software companies flagging new viruses and vulnerabilities.

The efforts to compromise security software were of particular importance because such software is relied upon to defend against an array of digital threats and is typically more trusted by the operating system than other applications, running with elevated privileges that allow more vectors for surveillance and attack. Spy agencies seem to be engaged in a digital game of cat and mouse with anti-virus software companies; the U.S. and U.K. have aggressively probed for weaknesses in software deployed by the companies, which have themselves exposed sophisticated state-sponsored malware.

The requested warrant, provided under Section 5 of the U.K.’s 1994 Intelligence Services Act, must be renewed by a government minister every six months. The document published today is a renewal request for a warrant valid from July 7, 2008 until January 7, 2009. The request seeks authorization for GCHQ activities that “involve modifying commercially available software to enable interception, decryption and other related tasks, or ‘reverse engineering’ software.”