Group items tagged

The Common Crawl Foundation’s repository of openly and freely accessible web crawl data is about to go live as a Public Data Set on Amazon Web Services.

Elbaz’ goal in developing the repository: “You can’t access, let alone download, the Google or the Bing crawl data. So certainly we’re differentiated in being very open and transparent about what we’re crawling and actually making it available to developers,” he says. “You might ask why is it going to be revolutionary to allow many more engineers and researchers and developers and students access to this data, whereas historically you have to work for one of the big search engines…. The question is, the world has the largest-ever corpus of knowledge out there on the web, and is there more that one can do with it than Google and Microsoft and a handful of other search engines are already doing? And the answer is unquestionably yes. ”

Common Crawl’s data already is stored on Amazon’s S3 service, but now Amazon will be providing the storage space for free through the Public Data Set program. Not only does that remove from Common Crawl the storage burden and costs for hosting its crawl of 5 billion web pages – some 50 or 60 terabytes large – but it should make it easier for users to access the data, and remove the bandwidth-related costs they might incur for downloads. Users won’t have to deal with setting up accounts, being responsible for bandwidth bills incurred, and more complex authentication processes.

On January 17, a three-judge panel of the U.S. Court of Appeals for the Ninth Circuit ruled, as a matter of first impression, that First Amendment defamation rules apply equally to both the institutional press and individual speakers and writers, such as bloggers.

In reaching this conclusion, the Ninth Circuit analyzed two key prior Supreme Court precedents: New York Times v. Sullivan (public official seeking damages for defamation must show “actual malice” as defined as a showing thatthe defendant published the defamatory statement with knowledge that it was false, or with reckless disregard as to whether it was false or not) and Gertz v. Robert Welch, Inc. (First Amendment requires only a negligence standard for private defamation actions). Notably, Gertz involved an institutional media defendant, and the Gertz Court invoked the need to shield “the press and broadcast media from the rigors of strict liability for defamation.” Yet neither New York Times nor Gertz, as the Ninth Circuit noted, were expressly limited to the institutional press. Moreover,a number of other Supreme Court cases have rejected such a limitation: Bartnicki v. Vopper; Cohen v. Cowles Media Co.; First National Bank of Boston v. Bellotti; and Citizens United v. Federal Election Commission.

On Feb. 25, the House of Representatives passed by a vote of 295-114 under suspension of the rules a bill aimed at creating a statutory right for owners of cellphones to be able to “unlock” their phones so that they can use the same phone with a different service provider.The Unlocking Consumer Choice Act (H.R. 1123), which was introduced in March by Rep. Robert W. Goodlatte (R-Va.), chairman of the House Judiciary Committee, was widely supported by members on both sides of the aisle.However, some representatives expressed objections to the current form of the legislation and even suggested that statutory protection of unlocking was no longer necessary, given that the Federal Communications Commission had in December persuaded the wireless industry to allow unlocking on a voluntary basis (241 PTD, 12/16/13).

On the morning of the day that the vote was to take place, several representatives who had previously supported the bill, issued a letter to their colleagues urging that H.R. 1123 be defeated on the floor of the House. The letter--signed by Reps. Zoe Lofgren (D-Calif.), Anna G. Eshoo (D-Calif.), Thomas H. Massie (R-Ky.), and Jared S. Polis (D-Colo.)--objected to a provision added to the bill after its approval by the full committee in July (148 PTD, 8/1/13).The new provision would exempt from protection “bulk unlocking” of phones. This provision might have something to with concerns expressed by some members of the Judiciary Committee in last year's hearings on the bill that permitting individual consumers to unlock their phones should not extend to businesses who charge consumers to unlock their phones for them.The letter referred to statements by the Electronic Frontier Foundation and Public Knowledge, consumer groups that had both supported the bill in the past, in which they withdrew their support because of the appearance of the new provision.

The Internet is always changing. Sites are rising and falling, content is deleted, and bad URLs can lead to '404 Not Found' errors that are as helpful as a brick wall. A new project proposes an do away with dead 404 errors by implementing new HTML code that will help access prior versions of hyperlinked content. With any luck, that means that you’ll never have to run into a dead link again. The “404-No-More” project is backed by a formidable coalition including members from organizations like the Harvard Library Innovation Lab, Los Alamos National Laboratory, Old Dominion University, and the Berkman Center for Internet & Society. Part of the Knight News Challenge, which seeks to strengthen the Internet for free expression and innovation through a variety of initiatives, 404-No-More recently reached the semifinal stage. The project aims to cure so-called link rot, the process by which hyperlinks become useless overtime because they point to addresses that are no longer available. If implemented, websites such as Wikipedia and other reference documents would be vastly improved. The new feature would also give Web authors a way provide links that contain both archived copies of content and specific dates of reference, the sort of information that diligent readers have to hunt down on a website like Archive.org.

While it may sound trivial, link rot can actually have real ramifications. Nearly 50 percent of the hyperlinks in Supreme Court decisions no longer work, a 2013 study revealed. Losing footnotes and citations in landmark legal decisions can mean losing crucial information and context about the laws that govern us. The same study found that 70 percent of URLs within the Harvard Law Review and similar journals didn’t link to the originally cited information, considered a serious loss surrounding the discussion of our laws. The project’s proponents have come up with more potential uses as well. Activists fighting censorship will have an easier time combatting government takedowns, for instance. Journalists will be much more capable of researching dynamic Web pages. “If every hyperlink was annotated with a publication date, you could automatically view an archived version of the content as the author intended for you to see it,” the project’s authors explain. The ephemeral nature of the Web could no longer be used as a weapon. Roger Macdonald, a director at the Internet Archive, called the 404-No-More project “an important contribution to preservation of knowledge.”

The new feature would come in the form of introducing the mset attribute to the <a> element in HTML, which would allow users of the code to specify multiple dates and copies of content as an external resource. For instance, if both the date of reference and the location of a copy of targeted content is known by an author, the new code would like like this: The 404-No-More project’s goals are numerous, but the ultimate goal is to have mset become a new HTML standard for hyperlinks. “An HTML standard that incorporates archives for hyperlinks will loop in these efforts and make the Web better for everyone,” project leaders wrote, “activists, journalists, and regular ol’ everyday web users.”

Huge volumes of private emails, phone calls, and internet chats are being intercepted by the National Security Agency with the secret cooperation of more foreign governments than previously known, according to newly disclosed documents from whistleblower Edward Snowden. The classified files, revealed today by the Danish newspaper Dagbladet Information in a reporting collaboration with The Intercept, shed light on how the NSA’s surveillance of global communications has expanded under a clandestine program, known as RAMPART-A, that depends on the participation of a growing network of intelligence agencies.

It has already been widely reported that the NSA works closely with eavesdropping agencies in the United Kingdom, Canada, New Zealand, and Australia as part of the so-called Five Eyes surveillance alliance. But the latest Snowden documents show that a number of other countries, described by the NSA as “third-party partners,” are playing an increasingly important role – by secretly allowing the NSA to install surveillance equipment on their fiber-optic cables. The NSA documents state that under RAMPART-A, foreign partners “provide access to cables and host U.S. equipment.” This allows the agency to covertly tap into “congestion points around the world” where it says it can intercept the content of phone calls, faxes, e-mails, internet chats, data from virtual private networks, and calls made using Voice over IP software like Skype.

The program, which the secret files show cost U.S. taxpayers about $170 million between 2011 and 2013, sweeps up a vast amount of communications at lightning speed. According to the intelligence community’s classified “Black Budget” for 2013, RAMPART-A enables the NSA to tap into three terabits of data every second as the data flows across the compromised cables – the equivalent of being able to download about 5,400 uncompressed high-definition movies every minute. In an emailed statement, the NSA declined to comment on the RAMPART-A program. “The fact that the U.S. government works with other nations, under specific and regulated conditions, mutually strengthens the security of all,” said NSA spokeswoman Vanee’ Vines. “NSA’s efforts are focused on ensuring the protection of the national security of the United States, its citizens, and our allies through the pursuit of valid foreign intelligence targets only.”