Skip to main content

Home/ Future of the Web/ Group items tagged law

Rss Feed Group items tagged

Gonzalo San Gil, PhD.

Copyright Law Is Being Rewritten Right Now, and You Can Help | WIRED - 0 views

  •  
    "Strap in, folks-because we're about to talk copyright law. I'm aware that as soon as I string the words "copyright" and "law" together, eyes start to glaze over. I get it." [# ! Let's take # ! ... the copyright back to its origins: # ! a tool to #promote #creation # ! instead of the actual use as # ! a #weapon to #mass #restriction.]
  •  
    "Strap in, folks-because we're about to talk copyright law. I'm aware that as soon as I string the words "copyright" and "law" together, eyes start to glaze over. I get it."
Gonzalo San Gil, PhD.

How Microsoft Appointed Itself Sheriff of the Internet | WIRED - 0 views

  •  
    "to disrupt the people behind this malicious software, most of which was running on the company's Windows operating system. So he used a controversial-but remarkably effective-legal maneuver that he invented himself. It's based on something called an ex parte temporary restraining order, and in conjunction with other laws such as the 1946 Lanham Act, it gives Microsoft the right to seize private assets-a power that typically lies within the purview of law enforcement, not private companies. "
  •  
    "to disrupt the people behind this malicious software, most of which was running on the company's Windows operating system. So he used a controversial-but remarkably effective-legal maneuver that he invented himself. It's based on something called an ex parte temporary restraining order, and in conjunction with other laws such as the 1946 Lanham Act, it gives Microsoft the right to seize private assets-a power that typically lies within the purview of law enforcement, not private companies. "
  •  
    "to disrupt the people behind this malicious software, most of which was running on the company's Windows operating system. So he used a controversial-but remarkably effective-legal maneuver that he invented himself. It's based on something called an ex parte temporary restraining order, and in conjunction with other laws such as the 1946 Lanham Act, it gives Microsoft the right to seize private assets-a power that typically lies within the purview of law enforcement, not private companies. "
Paul Merrell

Cy Vance's Proposal to Backdoor Encrypted Devices Is Riddled With Vulnerabilities | Jus... - 0 views

  • Less than a week after the attacks in Paris — while the public and policymakers were still reeling, and the investigation had barely gotten off the ground — Cy Vance, Manhattan’s District Attorney, released a policy paper calling for legislation requiring companies to provide the government with backdoor access to their smartphones and other mobile devices. This is the first concrete proposal of this type since September 2014, when FBI Director James Comey reignited the “Crypto Wars” in response to Apple’s and Google’s decisions to use default encryption on their smartphones. Though Comey seized on Apple’s and Google’s decisions to encrypt their devices by default, his concerns are primarily related to end-to-end encryption, which protects communications that are in transit. Vance’s proposal, on the other hand, is only concerned with device encryption, which protects data stored on phones. It is still unclear whether encryption played any role in the Paris attacks, though we do know that the attackers were using unencrypted SMS text messages on the night of the attack, and that some of them were even known to intelligence agencies and had previously been under surveillance. But regardless of whether encryption was used at some point during the planning of the attacks, as I lay out below, prohibiting companies from selling encrypted devices would not prevent criminals or terrorists from being able to access unbreakable encryption. Vance’s primary complaint is that Apple’s and Google’s decisions to provide their customers with more secure devices through encryption interferes with criminal investigations. He claims encryption prevents law enforcement from accessing stored data like iMessages, photos and videos, Internet search histories, and third party app data. He makes several arguments to justify his proposal to build backdoors into encrypted smartphones, but none of them hold water.
  • Before addressing the major privacy, security, and implementation concerns that his proposal raises, it is worth noting that while an increase in use of fully encrypted devices could interfere with some law enforcement investigations, it will help prevent far more crimes — especially smartphone theft, and the consequent potential for identity theft. According to Consumer Reports, in 2014 there were more than two million victims of smartphone theft, and nearly two-thirds of all smartphone users either took no steps to secure their phones or their data or failed to implement passcode access for their phones. Default encryption could reduce instances of theft because perpetrators would no longer be able to break into the phone to steal the data.
  • Vance argues that creating a weakness in encryption to allow law enforcement to access data stored on devices does not raise serious concerns for security and privacy, since in order to exploit the vulnerability one would need access to the actual device. He considers this an acceptable risk, claiming it would not be the same as creating a widespread vulnerability in encryption protecting communications in transit (like emails), and that it would be cheap and easy for companies to implement. But Vance seems to be underestimating the risks involved with his plan. It is increasingly important that smartphones and other devices are protected by the strongest encryption possible. Our devices and the apps on them contain astonishing amounts of personal information, so much that an unprecedented level of harm could be caused if a smartphone or device with an exploitable vulnerability is stolen, not least in the forms of identity fraud and credit card theft. We bank on our phones, and have access to credit card payments with services like Apple Pay. Our contact lists are stored on our phones, including phone numbers, emails, social media accounts, and addresses. Passwords are often stored on people’s phones. And phones and apps are often full of personal details about their lives, from food diaries to logs of favorite places to personal photographs. Symantec conducted a study, where the company spread 50 “lost” phones in public to see what people who picked up the phones would do with them. The company found that 95 percent of those people tried to access the phone, and while nearly 90 percent tried to access private information stored on the phone or in other private accounts such as banking services and email, only 50 percent attempted contacting the owner.
  • ...8 more annotations...
  • Vance attempts to downplay this serious risk by asserting that anyone can use the “Find My Phone” or Android Device Manager services that allow owners to delete the data on their phones if stolen. However, this does not stand up to scrutiny. These services are effective only when an owner realizes their phone is missing and can take swift action on another computer or device. This delay ensures some period of vulnerability. Encryption, on the other hand, protects everyone immediately and always. Additionally, Vance argues that it is safer to build backdoors into encrypted devices than it is to do so for encrypted communications in transit. It is true that there is a difference in the threats posed by the two types of encryption backdoors that are being debated. However, some manner of widespread vulnerability will inevitably result from a backdoor to encrypted devices. Indeed, the NSA and GCHQ reportedly hacked into a database to obtain cell phone SIM card encryption keys in order defeat the security protecting users’ communications and activities and to conduct surveillance. Clearly, the reality is that the threat of such a breach, whether from a hacker or a nation state actor, is very real. Even if companies go the extra mile and create a different means of access for every phone, such as a separate access key for each phone, significant vulnerabilities will be created. It would still be possible for a malicious actor to gain access to the database containing those keys, which would enable them to defeat the encryption on any smartphone they took possession of. Additionally, the cost of implementation and maintenance of such a complex system could be high.
  • Privacy is another concern that Vance dismisses too easily. Despite Vance’s arguments otherwise, building backdoors into device encryption undermines privacy. Our government does not impose a similar requirement in any other context. Police can enter homes with warrants, but there is no requirement that people record their conversations and interactions just in case they someday become useful in an investigation. The conversations that we once had through disposable letters and in-person conversations now happen over the Internet and on phones. Just because the medium has changed does not mean our right to privacy has.
  • In addition to his weak reasoning for why it would be feasible to create backdoors to encrypted devices without creating undue security risks or harming privacy, Vance makes several flawed policy-based arguments in favor of his proposal. He argues that criminals benefit from devices that are protected by strong encryption. That may be true, but strong encryption is also a critical tool used by billions of average people around the world every day to protect their transactions, communications, and private information. Lawyers, doctors, and journalists rely on encryption to protect their clients, patients, and sources. Government officials, from the President to the directors of the NSA and FBI, and members of Congress, depend on strong encryption for cybersecurity and data security. There are far more innocent Americans who benefit from strong encryption than there are criminals who exploit it. Encryption is also essential to our economy. Device manufacturers could suffer major economic losses if they are prohibited from competing with foreign manufacturers who offer more secure devices. Encryption also protects major companies from corporate and nation-state espionage. As more daily business activities are done on smartphones and other devices, they may now hold highly proprietary or sensitive information. Those devices could be targeted even more than they are now if all that has to be done to access that information is to steal an employee’s smartphone and exploit a vulnerability the manufacturer was required to create.
  • Vance also suggests that the US would be justified in creating such a requirement since other Western nations are contemplating requiring encryption backdoors as well. Regardless of whether other countries are debating similar proposals, we cannot afford a race to the bottom on cybersecurity. Heads of the intelligence community regularly warn that cybersecurity is the top threat to our national security. Strong encryption is our best defense against cyber threats, and following in the footsteps of other countries by weakening that critical tool would do incalculable harm. Furthermore, even if the US or other countries did implement such a proposal, criminals could gain access to devices with strong encryption through the black market. Thus, only innocent people would be negatively affected, and some of those innocent people might even become criminals simply by trying to protect their privacy by securing their data and devices. Finally, Vance argues that David Kaye, UN Special Rapporteur for Freedom of Expression and Opinion, supported the idea that court-ordered decryption doesn’t violate human rights, provided certain criteria are met, in his report on the topic. However, in the context of Vance’s proposal, this seems to conflate the concepts of court-ordered decryption and of government-mandated encryption backdoors. The Kaye report was unequivocal about the importance of encryption for free speech and human rights. The report concluded that:
  • States should promote strong encryption and anonymity. National laws should recognize that individuals are free to protect the privacy of their digital communications by using encryption technology and tools that allow anonymity online. … States should not restrict encryption and anonymity, which facilitate and often enable the rights to freedom of opinion and expression. Blanket prohibitions fail to be necessary and proportionate. States should avoid all measures that weaken the security that individuals may enjoy online, such as backdoors, weak encryption standards and key escrows. Additionally, the group of intelligence experts that was hand-picked by the President to issue a report and recommendations on surveillance and technology, concluded that: [R]egarding encryption, the U.S. Government should: (1) fully support and not undermine efforts to create encryption standards; (2) not in any way subvert, undermine, weaken, or make vulnerable generally available commercial software; and (3) increase the use of encryption and urge US companies to do so, in order to better protect data in transit, at rest, in the cloud, and in other storage.
  • The clear consensus among human rights experts and several high-ranking intelligence experts, including the former directors of the NSA, Office of the Director of National Intelligence, and DHS, is that mandating encryption backdoors is dangerous. Unaddressed Concerns: Preventing Encrypted Devices from Entering the US and the Slippery Slope In addition to the significant faults in Vance’s arguments in favor of his proposal, he fails to address the question of how such a restriction would be effectively implemented. There is no effective mechanism for preventing code from becoming available for download online, even if it is illegal. One critical issue the Vance proposal fails to address is how the government would prevent, or even identify, encrypted smartphones when individuals bring them into the United States. DHS would have to train customs agents to search the contents of every person’s phone in order to identify whether it is encrypted, and then confiscate the phones that are. Legal and policy considerations aside, this kind of policy is, at the very least, impractical. Preventing strong encryption from entering the US is not like preventing guns or drugs from entering the country — encrypted phones aren’t immediately obvious as is contraband. Millions of people use encrypted devices, and tens of millions more devices are shipped to and sold in the US each year.
  • Finally, there is a real concern that if Vance’s proposal were accepted, it would be the first step down a slippery slope. Right now, his proposal only calls for access to smartphones and devices running mobile operating systems. While this policy in and of itself would cover a number of commonplace devices, it may eventually be expanded to cover laptop and desktop computers, as well as communications in transit. The expansion of this kind of policy is even more worrisome when taking into account the speed at which technology evolves and becomes widely adopted. Ten years ago, the iPhone did not even exist. Who is to say what technology will be commonplace in 10 or 20 years that is not even around today. There is a very real question about how far law enforcement will go to gain access to information. Things that once seemed like merely science fiction, such as wearable technology and artificial intelligence that could be implanted in and work with the human nervous system, are now available. If and when there comes a time when our “smart phone” is not really a device at all, but is rather an implant, surely we would not grant law enforcement access to our minds.
  • Policymakers should dismiss Vance’s proposal to prohibit the use of strong encryption to protect our smartphones and devices in order to ensure law enforcement access. Undermining encryption, regardless of whether it is protecting data in transit or at rest, would take us down a dangerous and harmful path. Instead, law enforcement and the intelligence community should be working to alter their skills and tactics in a fast-evolving technological world so that they are not so dependent on information that will increasingly be protected by encryption.
Paul Merrell

The All Writs Act, Software Licenses, and Why Judges Should Ask More Questions | Just S... - 0 views

  • Pending before federal magistrate judge James Orenstein is the government’s request for an order obligating Apple, Inc. to unlock an iPhone and thereby assist prosecutors in decrypting data the government has seized and is authorized to search pursuant to a warrant. In an order questioning the government’s purported legal basis for this request, the All Writs Act of 1789 (AWA), Judge Orenstein asked Apple for a brief informing the court whether the request would be technically feasible and/or burdensome. After Apple filed, the court asked it to file a brief discussing whether the government had legal grounds under the AWA to compel Apple’s assistance. Apple filed that brief and the government filed a reply brief last week in the lead-up to a hearing this morning.
  • We’ve long been concerned about whether end users own software under the law. Software owners have rights of adaptation and first sale enshrined in copyright law. But software publishers have claimed that end users are merely licensees, and our rights under copyright law can be waived by mass-market end user license agreements, or EULAs. Over the years, Granick has argued that users should retain their rights even if mass-market licenses purport to take them away. The government’s brief takes advantage of Apple’s EULA for iOS to argue that Apple, the software publisher, is responsible for iPhones around the world. Apple’s EULA states that when you buy an iPhone, you’re not buying the iOS software it runs, you’re just licensing it from Apple. The government argues that having designed a passcode feature into a copy of software which it owns and licenses rather than sells, Apple can be compelled under the All Writs Act to bypass the passcode on a defendant’s iPhone pursuant to a search warrant and thereby access the software owned by Apple. Apple’s supplemental brief argues that in defining its users’ contractual rights vis-à-vis Apple with regard to Apple’s intellectual property, Apple in no way waived its own due process rights vis-à-vis the government with regard to users’ devices. Apple’s brief compares this argument to forcing a car manufacturer to “provide law enforcement with access to the vehicle or to alter its functionality at the government’s request” merely because the car contains licensed software. 
  • This is an interesting twist on the decades-long EULA versus users’ rights fight. As far as we know, this is the first time that the government has piggybacked on EULAs to try to compel software companies to provide assistance to law enforcement. Under the government’s interpretation of the All Writs Act, anyone who makes software could be dragooned into assisting the government in investigating users of the software. If the court adopts this view, it would give investigators immense power. The quotidian aspects of our lives increasingly involve software (from our cars to our TVs to our health to our home appliances), and most of that software is arguably licensed, not bought. Conscripting software makers to collect information on us would afford the government access to the most intimate information about us, on the strength of some words in some license agreements that people never read. (And no wonder: The iPhone’s EULA came to over 300 pages when the government filed it as an exhibit to its brief.)
  • ...1 more annotation...
  • The government’s brief does not acknowledge the sweeping implications of its arguments. It tries to portray its requested unlocking order as narrow and modest, because it “would not require Apple to make any changes to its software or hardware, … [or] to introduce any new ability to access data on its phones. It would simply require Apple to use its existing capability to bypass the passcode on a passcode-locked iOS 7 phone[.]” But that undersells the implications of the legal argument the government is making: that anything a company already can do, it could be compelled to do under the All Writs Act in order to assist law enforcement. Were that the law, the blow to users’ trust in their encrypted devices, services, and products would be little different than if Apple and other companies were legally required to design backdoors into their encryption mechanisms (an idea the government just can’t seem to drop, its assurances in this brief notwithstanding). Entities around the world won’t buy security software if its makers cannot be trusted not to hand over their users’ secrets to the US government. That’s what makes the encryption in iOS 8 and later versions, which Apple has told the court it “would not have the technical ability” to bypass, so powerful — and so despised by the government: Because no matter how broadly the All Writs Act extends, no court can compel Apple to do the impossible.
Paul Merrell

Big Tech companies appeal to Supreme Court to strike down Texas law banning political c... - 0 views

  • Facebook and Google are among multiple Big Tech companies seeking to have Texas’ new law banning political censorship on social media axed by the Supreme Court. According to The Washington Post, advocacy groups NetChoice and the Computer & Communications Industry Association (CCIA) filed an emergency application with the U.S. Supreme Court on Friday on behalf of Facebook, Google and other Big Tech companies, with the intention of striking down the new Texas law that prohibits censorship based on political ideology on social media. In a statement about the emergency filing, NetChoice counsel Chris Marchese argues that the Texas law, which went into effect last Wednesday, “strips private online businesses of their speech rights, forbids them from making constitutionally protected editorial decisions, and forces them to publish and promote objectionable content.”
  • “Left standing, [the Texas law] will turn the First Amendment on its head — to violate free speech, the government need only claim to be ‘protecting’ it,” Marchese added. Under the law, Texas residents and the state’s attorney general would be permitted to sue social media companies based in the United States if they believe their social media accounts were censored based on their political views. While the law was initially blocked by a federal district judge after it was signed last September by Texas Gov. Greg Abbot, the injunction was ultimately lifted by an appeals court last Wednesday. Big Tech companies have been consistently charged with weaponizing their “Terms of Service” agreements with users in an effort to ban or censor those expressing traditionally conservative or right-wing views on their extremely large and influential platforms.
Paul Merrell

Shaking My Head - Medium - 0 views

  • Last month, at the request of the Department of Justice, the Courts approved changes to the obscure Rule 41 of the Federal Rules of Criminal Procedure, which governs search and seizure. By the nature of this obscure bureaucratic process, these rules become law unless Congress rejects the changes before December 1, 2016.Today I, along with my colleagues Senators Paul from Kentucky, Baldwin from Wisconsin, and Daines and Tester from Montana, am introducing the Stopping Mass Hacking (SMH) Act (bill, summary), a bill to protect millions of law-abiding Americans from a massive expansion of government hacking and surveillance. Join the conversation with #SMHact.
  • For law enforcement to conduct a remote electronic search, they generally need to plant malware in — i.e. hack — a device. These rule changes will allow the government to search millions of computers with the warrant of a single judge. To me, that’s clearly a policy change that’s outside the scope of an “administrative change,” and it is something that Congress should consider. An agency with the record of the Justice Department shouldn’t be able to wave its arms and grant itself entirely new powers.
  • These changes say that if law enforcement doesn’t know where an electronic device is located, a magistrate judge will now have the the authority to issue a warrant to remotely search the device, anywhere in the world. While it may be appropriate to address the issue of allowing a remote electronic search for a device at an unknown location, Congress needs to consider what protections must be in place to protect Americans’ digital security and privacy. This is a new and uncertain area of law, so there needs to be full and careful debate. The ACLU has a thorough discussion of the Fourth Amendment ramifications and the technological questions at issue with these kinds of searches.The second part of the change to Rule 41 would give a magistrate judge the authority to issue a single warrant that would authorize the search of an unlimited number — potentially thousands or millions — of devices, located anywhere in the world. These changes would dramatically expand the government’s hacking and surveillance authority. The American public should understand that these changes won’t just affect criminals: computer security experts and civil liberties advocates say the amendments would also dramatically expand the government’s ability to hack the electronic devices of law-abiding Americans if their devices were affected by a computer attack. Devices will be subject to search if their owners were victims of a botnet attack — so the government will be treating victims of hacking the same way they treat the perpetrators.
  • ...1 more annotation...
  • As the Center on Democracy and Technology has noted, there are approximately 500 million computers that fall under this rule. The public doesn’t know nearly enough about how law enforcement executes these hacks, and what risks these types of searches will pose. By compromising the computer’s system, the search might leave it open to other attackers or damage the computer they are searching.Don’t take it from me that this will impact your security, read more from security researchers Steven Bellovin, Matt Blaze and Susan Landau.Finally, these changes to Rule 41 would also give some types of electronic searches different, weaker notification requirements than physical searches. Under this new Rule, they are only required to make “reasonable efforts” to notify people that their computers were searched. This raises the possibility of the FBI hacking into a cyber attack victim’s computer and not telling them about it until afterward, if at all.
Gonzalo San Gil, PhD.

Patent abuse litigation laws passed or pending in over twenty U.S. states | Opensource.com - 0 views

  •  
    "Patent reform may have stalled this year at the federal level, but patent trolls may soon find their actions curtailed by a number of patent abuse litigation laws that have been passed or are pending in over twenty U.S. states."
  •  
    "Patent reform may have stalled this year at the federal level, but patent trolls may soon find their actions curtailed by a number of patent abuse litigation laws that have been passed or are pending in over twenty U.S. states."
Paul Merrell

The Fundamentals of US Surveillance: What Edward Snowden Never Told Us? | Global Resear... - 0 views

  • Former US intelligence contractor Edward Snowden’s revelations rocked the world.  According to his detailed reports, the US had launched massive spying programs and was scrutinizing the communications of American citizens in a manner which could only be described as extreme and intense. The US’s reaction was swift and to the point. “”Nobody is listening to your telephone calls,” President Obama said when asked about the NSA. As quoted in The Guardian,  Obama went on to say that surveillance programs were “fully overseen not just by Congress but by the Fisa court, a court specially put together to evaluate classified programs to make sure that the executive branch, or government generally, is not abusing them”. However, it appears that Snowden may have missed a pivotal part of the US surveillance program. And in stating that the “nobody” is not listening to our calls, President Obama may have been fudging quite a bit.
  • In fact, Great Britain maintains a “listening post” at NSA HQ. The laws restricting live wiretaps do not apply to foreign countries  and thus this listening post  is not subject to  US law.  In other words, the restrictions upon wiretaps, etc. do not apply to the British listening post.  So when Great Britain hands over the recordings to the NSA, technically speaking, a law is not being broken and technically speaking, the US is not eavesdropping on our each and every call. It is Great Britain which is doing the eavesdropping and turning over these records to US intelligence. According to John Loftus, formerly an attorney with  the Department of Justice and author of a number of books concerning US intelligence activities, back in the late seventies  the USDOJ issued a memorandum proposing an amendment to FISA. Loftus, who recalls seeing  the memo, stated in conversation this week that the DOJ proposed inserting the words “by the NSA” into the FISA law  so the scope of the law would only restrict surveillance by the NSA, not by the British.  Any subsequent sharing of the data culled through the listening posts was strictly outside the arena of FISA. Obama was less than forthcoming when he insisted that “What I can say unequivocally is that if you are a US person, the NSA cannot listen to your telephone calls, and the NSA cannot target your emails … and have not.”
  • According to Loftus, the NSA is indeed listening as Great Britain is turning over the surveillance records en masse to that agency. Loftus states that the arrangement is reciprocal, with the US maintaining a parallel listening post in Great Britain. In an interview this past week, Loftus told this reporter that  he believes that Snowden simply did not know about the arrangement between Britain and the US. As a contractor, said Loftus, Snowden would not have had access to this information and thus his detailed reports on the extent of US spying, including such programs as XKeyscore, which analyzes internet data based on global demographics, and PRISM, under which the telecommunications companies, such as Google, Facebook, et al, are mandated to collect our communications, missed the critical issue of the FISA loophole.
  • ...2 more annotations...
  • U.S. government officials have defended the program by asserting it cannot be used on domestic targets without a warrant. But once again, the FISA courts and their super-secret warrants  do not apply to foreign government surveillance of US citizens. So all this sturm and drang about whether or not the US is eavesdropping on our communications is, in fact, irrelevant and diversionary.
  • In fact, the USA Freedom Act reinstituted a number of the surveillance protocols of Section 215, including  authorization for  roving wiretaps  and tracking “lone wolf terrorists.”  While mainstream media heralded the passage of the bill as restoring privacy rights which were shredded under 215, privacy advocates have maintained that the bill will do little, if anything, to reverse the  surveillance situation in the US. The NSA went on the record as supporting the Freedom Act, stating it would end bulk collection of telephone metadata. However, in light of the reciprocal agreement between the US and Great Britain, the entire hoopla over NSA surveillance, Section 215, FISA courts and the USA Freedom Act could be seen as a giant smokescreen. If Great Britain is collecting our real time phone conversations and turning them over to the NSA, outside the realm or reach of the above stated laws, then all this posturing over the privacy rights of US citizens and surveillance laws expiring and being resurrected doesn’t amount to a hill of CDs.
Paul Merrell

Vodafone reveals existence of secret wires that allow state surveillance | Business | T... - 0 views

  • Vodafone, one of the world's largest mobile phone groups, has revealed the existence of secret wires that allow government agencies to listen to all conversations on its networks, saying they are widely used in some of the 29 countries in which it operates in Europe and beyond.The company has broken its silence on government surveillance in order to push back against the increasingly widespread use of phone and broadband networks to spy on citizens, and will publish its first Law Enforcement Disclosure Report on Friday. At 40,000 words, it is the most comprehensive survey yet of how governments monitor the conversations and whereabouts of their people.The company said wires had been connected directly to its network and those of other telecoms groups, allowing agencies to listen to or record live conversations and, in certain cases, track the whereabouts of a customer. Privacy campaigners said the revelations were a "nightmare scenario" that confirmed their worst fears on the extent of snooping.
  • Vodafone's group privacy officer, Stephen Deadman, said: "These pipes exist, the direct access model exists."We are making a call to end direct access as a means of government agencies obtaining people's communication data. Without an official warrant, there is no external visibility. If we receive a demand we can push back against the agency. The fact that a government has to issue a piece of paper is an important constraint on how powers are used."Vodafone is calling for all direct-access pipes to be disconnected, and for the laws that make them legal to be amended. It says governments should "discourage agencies and authorities from seeking direct access to an operator's communications infrastructure without a lawful mandate".
  • In America, Verizon and AT&T have published data, but only on their domestic operations. Deutsche Telekom in Germany and Telstra in Australia have also broken ground at home. Vodafone is the first to produce a global survey.
  • ...2 more annotations...
  • Peter Micek, policy counsel at the campaign group Access, said: "In a sector that has historically been quiet about how it facilitates government access to user data, Vodafone has for the first time shone a bright light on the challenges of a global telecom giant, giving users a greater understanding of the demands governments make of telcos. Vodafone's report also highlights how few governments issue any transparency reports, with little to no information about the number of wiretaps, cell site tower dumps, and other invasive surveillance practices."
  • Snowden, the National Security Agency whistleblower, joined Google, Reddit, Mozilla and other tech firms and privacy groups on Thursday to call for a strengthening of privacy rights online in a "Reset the net" campaign.Twelve months after revelations about the scale of the US government's surveillance programs were first published in the Guardian and the Washington Post, Snowden said: "One year ago, we learned that the internet is under surveillance, and our activities are being monitored to create permanent records of our private lives – no matter how innocent or ordinary those lives might be. Today, we can begin the work of effectively shutting down the collection of our online communications, even if the US Congress fails to do the same."
  •  
    The Vodafone disclosures will undoubtedly have a very large ripple effect. Note carefully that this is the first major telephone service in the world to break ranks with the others and come out swinging at secret government voyeur agencies. Will others follow. If you follow the links to the Vodafone report, you'll find a very handy big PDF providing an overview of the relevant laws in each of the customer nations. There's a cute Guardian table that shows the aggregate number of warrants for interception of content via Vodafone for each of those nations, broken down by content type. That table has white-on-black cells noting where disclosure of those types of surveillance statistics are prohibited by law. So it is far from a complete picture, but it's a heck of a good start.  But several of those customer nations are members of the E.U., where digital privacy rights are enshrined as human rights under an EU-wide treaty. So expect some heat to roll downhill on those nations from the European treaty organizations, particularly the European Court of Human Rights, staffed with civil libertarian judges, from which there is no appeal.     
Paul Merrell

UN Report Finds Mass Surveillance Violates International Treaties and Privacy Rights - ... - 0 views

  • The United Nations’ top official for counter-terrorism and human rights (known as the “Special Rapporteur”) issued a formal report to the U.N. General Assembly today that condemns mass electronic surveillance as a clear violation of core privacy rights guaranteed by multiple treaties and conventions. “The hard truth is that the use of mass surveillance technology effectively does away with the right to privacy of communications on the Internet altogether,” the report concluded. Central to the Rapporteur’s findings is the distinction between “targeted surveillance” — which “depend[s] upon the existence of prior suspicion of the targeted individual or organization” — and “mass surveillance,” whereby “states with high levels of Internet penetration can [] gain access to the telephone and e-mail content of an effectively unlimited number of users and maintain an overview of Internet activity associated with particular websites.” In a system of “mass surveillance,” the report explained, “all of this is possible without any prior suspicion related to a specific individual or organization. The communications of literally every Internet user are potentially open for inspection by intelligence and law enforcement agencies in the States concerned.”
  • Mass surveillance thus “amounts to a systematic interference with the right to respect for the privacy of communications,” it declared. As a result, “it is incompatible with existing concepts of privacy for States to collect all communications or metadata all the time indiscriminately.” In concluding that mass surveillance impinges core privacy rights, the report was primarily focused on the International Covenant on Civil and Political Rights, a treaty enacted by the General Assembly in 1966, to which all of the members of the “Five Eyes” alliance are signatories. The U.S. ratified the treaty in 1992, albeit with various reservations that allowed for the continuation of the death penalty and which rendered its domestic law supreme. With the exception of the U.S.’s Persian Gulf allies (Saudi Arabia, UAE and Qatar), virtually every major country has signed the treaty. Article 17 of the Covenant guarantees the right of privacy, the defining protection of which, the report explained, is “that individuals have the right to share information and ideas with one another without interference by the State, secure in the knowledge that their communication will reach and be read by the intended recipients alone.”
  • The report’s key conclusion is that this core right is impinged by mass surveillance programs: “Bulk access technology is indiscriminately corrosive of online privacy and impinges on the very essence of the right guaranteed by article 17. In the absence of a formal derogation from States’ obligations under the Covenant, these programs pose a direct and ongoing challenge to an established norm of international law.” The report recognized that protecting citizens from terrorism attacks is a vital duty of every state, and that the right of privacy is not absolute, as it can be compromised when doing so is “necessary” to serve “compelling” purposes. It noted: “There may be a compelling counter-terrorism justification for the radical re-evaluation of Internet privacy rights that these practices necessitate. ” But the report was adamant that no such justifications have ever been demonstrated by any member state using mass surveillance: “The States engaging in mass surveillance have so far failed to provide a detailed and evidence-based public justification for its necessity, and almost no States have enacted explicit domestic legislation to authorize its use.”
  • ...5 more annotations...
  • Instead, explained the Rapporteur, states have relied on vague claims whose validity cannot be assessed because of the secrecy behind which these programs are hidden: “The arguments in favor of a complete abrogation of the right to privacy on the Internet have not been made publicly by the States concerned or subjected to informed scrutiny and debate.” About the ongoing secrecy surrounding the programs, the report explained that “states deploying this technology retain a monopoly of information about its impact,” which is “a form of conceptual censorship … that precludes informed debate.” A June report from the High Commissioner for Human Rights similarly noted “the disturbing lack of governmental transparency associated with surveillance policies, laws and practices, which hinders any effort to assess their coherence with international human rights law and to ensure accountability.” The rejection of the “terrorism” justification for mass surveillance as devoid of evidence echoes virtually every other formal investigation into these programs. A federal judge last December found that the U.S. Government was unable to “cite a single case in which analysis of the NSA’s bulk metadata collection actually stopped an imminent terrorist attack.” Later that month, President Obama’s own Review Group on Intelligence and Communications Technologies concluded that mass surveillance “was not essential to preventing attacks” and information used to detect plots “could readily have been obtained in a timely manner using conventional [court] orders.”
  • That principle — that the right of internet privacy belongs to all individuals, not just Americans — was invoked by NSA whistleblower Edward Snowden when he explained in a June, 2013 interview at The Guardian why he disclosed documents showing global surveillance rather than just the surveillance of Americans: “More fundamentally, the ‘US Persons’ protection in general is a distraction from the power and danger of this system. Suspicionless surveillance does not become okay simply because it’s only victimizing 95% of the world instead of 100%.” The U.N. Rapporteur was clear that these systematic privacy violations are the result of a union between governments and tech corporations: “States increasingly rely on the private sector to facilitate digital surveillance. This is not confined to the enactment of mandatory data retention legislation. Corporates [sic] have also been directly complicit in operationalizing bulk access technology through the design of communications infrastructure that facilitates mass surveillance. ”
  • The report was most scathing in its rejection of a key argument often made by American defenders of the NSA: that mass surveillance is justified because Americans are given special protections (the requirement of a FISA court order for targeted surveillance) which non-Americans (95% of the world) do not enjoy. Not only does this scheme fail to render mass surveillance legal, but it itself constitutes a separate violation of international treaties (emphasis added): The Special Rapporteur concurs with the High Commissioner for Human Rights that where States penetrate infrastructure located outside their territorial jurisdiction, they remain bound by their obligations under the Covenant. Moreover, article 26 of the Covenant prohibits discrimination on grounds of, inter alia, nationality and citizenship. The Special Rapporteur thus considers that States are legally obliged to afford the same privacy protection for nationals and non-nationals and for those within and outside their jurisdiction. Asymmetrical privacy protection regimes are a clear violation of the requirements of the Covenant.
  • Three Democratic Senators on the Senate Intelligence Committee wrote in The New York Times that “the usefulness of the bulk collection program has been greatly exaggerated” and “we have yet to see any proof that it provides real, unique value in protecting national security.” A study by the centrist New America Foundation found that mass metadata collection “has had no discernible impact on preventing acts of terrorism” and, where plots were disrupted, “traditional law enforcement and investigative methods provided the tip or evidence to initiate the case.” It labeled the NSA’s claims to the contrary as “overblown and even misleading.” While worthless in counter-terrorism policies, the UN report warned that allowing mass surveillance to persist with no transparency creates “an ever present danger of ‘purpose creep,’ by which measures justified on counter-terrorism grounds are made available for use by public authorities for much less weighty public interest purposes.” Citing the UK as one example, the report warned that, already, “a wide range of public bodies have access to communications data, for a wide variety of purposes, often without judicial authorization or meaningful independent oversight.”
  • The latest finding adds to the growing number of international formal rulings that the mass surveillance programs of the U.S. and its partners are illegal. In January, the European parliament’s civil liberties committee condemned such programs in “the strongest possible terms.” In April, the European Court of Justice ruled that European legislation on data retention contravened EU privacy rights. A top secret memo from the GCHQ, published last year by The Guardian, explicitly stated that one key reason for concealing these programs was fear of a “damaging public debate” and specifically “legal challenges against the current regime.” The report ended with a call for far greater transparency along with new protections for privacy in the digital age. Continuation of the status quo, it warned, imposes “a risk that systematic interference with the security of digital communications will continue to proliferate without any serious consideration being given to the implications of the wholesale abandonment of the right to online privacy.” The urgency of these reforms is underscored, explained the Rapporteur, by a conclusion of the United States Privacy and Civil Liberties Oversight Board that “permitting the government to routinely collect the calling records of the entire nation fundamentally shifts the balance of power between the state and its citizens.”
Paul Merrell

​EU admonishes US for overseas data requests - RT News - 0 views

  • The EU has slammed the US for its demand that Microsoft surrender overseas data – emails held on Irish servers – saying that the move could contravene international law. The US attempt to make Microsoft provide the emails prompted Viviane Reding, vice-president of the European Commission, to offer support to Microsoft and openly criticize the loss of personal information it could potentially involve. “The commission’s concern is that the extraterritorial application of foreign laws [and orders to companies based thereon] may be in breach of international law,” Reding wrote last week in a letter responding to questions from Dutch MEP Sophia in't Veld, reported the Financial Times on Monday. The move would “hurt the competitiveness of US cloud providers in general,” Microsoft said, adding that: “Microsoft and US technology companies have faced growing mistrust and concern about their ability to protect the privacy of personal information located outside the US.”
  • Reding added that the US “may impede the attainment of the protection of individuals guaranteed” under EU law. Her statement further echoes arguments laid out by Apple, Cisco, AT&T, and Verizon, which supported Microsoft against the US warrant. At the beginning of June, Microsoft compared the warrant to an authorization for federal agents ‘to break down the doors’ of its Dublin facility. Reding said the US should have leaned away from coercion and instead depended on mutual legal assistance treaties that facilitate law enforcement agency cooperation.
  • “Companies bound by EU data protection law who receive such a court order are caught in the middle of such situations where there is, as you say in your letter, a conflict of laws,” Reding wrote.
Paul Merrell

Victory for Users: Librarian of Congress Renews and Expands Protections for Fair Uses |... - 0 views

  • The new rules for exemptions to copyright's DRM-circumvention laws were issued today, and the Librarian of Congress has granted much of what EFF asked for over the course of months of extensive briefs and hearings. The exemptions we requested—ripping DVDs and Blurays for making fair use remixes and analysis; preserving video games and running multiplayer servers after publishers have abandoned them; jailbreaking cell phones, tablets, and other portable computing devices to run third party software; and security research and modification and repairs on cars—have each been accepted, subject to some important caveats.
  • The exemptions are needed thanks to a fundamentally flawed law that forbids users from breaking DRM, even if the purpose is a clearly lawful fair use. As software has become ubiquitous, so has DRM.  Users often have to circumvent that DRM to make full use of their devices, from DVDs to games to smartphones and cars. The law allows users to request exemptions for such lawful uses—but it doesn’t make it easy. Exemptions are granted through an elaborate rulemaking process that takes place every three years and places a heavy burden on EFF and the many other requesters who take part. Every exemption must be argued anew, even if it was previously granted, and even if there is no opposition. The exemptions that emerge are limited in scope. What is worse, they only apply to end users—the people who are actually doing the ripping, tinkering, jailbreaking, or research—and not to the people who make the tools that facilitate those lawful activities. The section of the law that creates these restrictions—the Digital Millennium Copyright Act's Section 1201—is fundamentally flawed, has resulted in myriad unintended consequences, and is long past due for reform or removal altogether from the statute books. Still, as long as its rulemaking process exists, we're pleased to have secured the following exemptions.
  • The new rules are long and complicated, and we'll be posting more details about each as we get a chance to analyze them. In the meantime, we hope each of these exemptions enable more exciting fair uses that educate, entertain, improve the underlying technology, and keep us safer. A better long-terms solution, though, is to eliminate the need for this onerous rulemaking process. We encourage lawmakers to support efforts like the Unlocking Technology Act, which would limit the scope of Section 1201 to copyright infringements—not fair uses. And as the White House looks for the next Librarian of Congress, who is ultimately responsible for issuing the exemptions, we hope to get a candidate who acts—as a librarian should—in the interest of the public's access to information.
Gonzalo San Gil, PhD.

"#PiracyHarms" Are Now Part of U.S. #Education #Law | TorrentFreak | # ! What... - 0 views

  •  
    | Ernesto | Last month President Barack Obama signed the Every Student Succeeds Act into law, making $1 billion dollars available for educational technology spending. In addition, the new law ensures that educators are aware of the piracy harms new technologies introduce.
Paul Merrell

White House, Intel Chiefs Want To Make Digital Spying Law Permanent | HuffPost - 0 views

  • The White House and U.S. intelligence chiefs Wednesday backed making permanent a law that allows for the collection of digital communications of foreigners overseas, escalating a fight in Congress over privacy and security. The law, enshrined in Section 702 of the Foreign Intelligence Surveillance Act, is due to expire on December 31 unless Congress votes to reauthorize it, but is considered vital by U.S. intelligence agencies. Privacy advocates have criticized the law though for allowing the incidental collection of data belonging to millions of Americans without a search warrant. The push to make the law permanent may lead to a contentious debate over renewal of Section 702 in Congress, where lawmakers in both parties are deeply divided over whether to adopt transparency and oversight reforms
  • Reuters reported in March that the Trump administration supported renewal of Section 702 without any changes, citing an unnamed White House official, but it was not clear at the time whether it wanted the law made permanent.
Paul Merrell

Long-Secret Stingray Manuals Detail How Police Can Spy on Phones - 0 views

  • Harris Corp.’s Stingray surveillance device has been one of the most closely guarded secrets in law enforcement for more than 15 years. The company and its police clients across the United States have fought to keep information about the mobile phone-monitoring boxes from the public against which they are used. The Intercept has obtained several Harris instruction manuals spanning roughly 200 pages and meticulously detailing how to create a cellular surveillance dragnet. Harris has fought to keep its surveillance equipment, which carries price tags in the low six figures, hidden from both privacy activists and the general public, arguing that information about the gear could help criminals. Accordingly, an older Stingray manual released under the Freedom of Information Act to news website TheBlot.com last year was almost completely redacted. So too have law enforcement agencies at every level, across the country, evaded almost all attempts to learn how and why these extremely powerful tools are being used — though court battles have made it clear Stingrays are often deployed without any warrant. The San Bernardino Sheriff’s Department alone has snooped via Stingray, sans warrant, over 300 times.
  • The documents described and linked below, instruction manuals for the software used by Stingray operators, were provided to The Intercept as part of a larger cache believed to have originated with the Florida Department of Law Enforcement. Two of them contain a “distribution warning” saying they contain “Proprietary Information and the release of this document and the information contained herein is prohibited to the fullest extent allowable by law.”  Although “Stingray” has become a catch-all name for devices of its kind, often referred to as “IMSI catchers,” the manuals include instructions for a range of other Harris surveillance boxes, including the Hailstorm, ArrowHead, AmberJack, and KingFish. They make clear the capability of those devices and the Stingray II to spy on cellphones by, at minimum, tracking their connection to the simulated tower, information about their location, and certain “over the air” electronic messages sent to and from them. Wessler added that parts of the manuals make specific reference to permanently storing this data, something that American law enforcement has denied doing in the past.
  • One piece of Windows software used to control Harris’s spy boxes, software that appears to be sold under the name “Gemini,” allows police to track phones across 2G, 3G, and LTE networks. Another Harris app, “iDen Controller,” provides a litany of fine-grained options for tracking phones. A law enforcement agent using these pieces of software along with Harris hardware could not only track a large number of phones as they moved throughout a city but could also apply nicknames to certain phones to keep track of them in the future. The manual describing how to operate iDEN, the lengthiest document of the four at 156 pages, uses an example of a target (called a “subscriber”) tagged alternately as Green Boy and Green Ben:
  • ...2 more annotations...
  • In order to maintain an uninterrupted connection to a target’s phone, the Harris software also offers the option of intentionally degrading (or “redirecting”) someone’s phone onto an inferior network, for example, knocking a connection from LTE to 2G:
  • A video of the Gemini software installed on a personal computer, obtained by The Intercept and embedded below, provides not only an extensive demonstration of the app but also underlines how accessible the mass surveillance code can be: Installing a complete warrantless surveillance suite is no more complicated than installing Skype. Indeed, software such as Photoshop or Microsoft Office, which require a registration key or some other proof of ownership, are more strictly controlled by their makers than software designed for cellular interception.
Gonzalo San Gil, PhD.

International Principles on the Application of Human Rights to Communications Surveilla... - 1 views

  •  
    "Final version 10 July 2013 As technologies that facilitate State surveillance of communications advance, States are failing to ensure that laws and regulations related to communications surveillance adhere to international human rights and adequately protect the rights to privacy and freedom of expression. This document attempts to explain how international human rights law applies in the current digital environment, particularly in light of the increase in and changes to communications surveillance technologies and techniques. These principles can provide civil society groups, industry, States and others with a framework to evaluate whether current or proposed surveillance laws and practices are consistent with human rights."
Gonzalo San Gil, PhD.

EU high court strikes down metadata collection law | Ars Technica - 1 views

  •  
    "Citizens made to feel that they "are the subject of constant surveillance." by Cyrus Farivar - Apr 8 2014, 4:25pm CEST"
  •  
    Just finished reading the court's opinion. I can only wish that the U.S. government had such fine-tuned respect for civil rights Not quoted in the linked article, but opinion paragraph 68 is very bad news for U.S. service providers: "In the second place, it should be added that that directive does not require the data in question to be retained within the European Union, with the result that it cannot be held that the control, explicitly required by Article 8(3) of the Charter, by an independent authority of compliance with the requirements of protection and security, as referred to in the two previous paragraphs, is fully ensured. Such a control, carried out on the basis of EU law, is an essential component of the protection of individuals with regard to the processing of personal data (see, to that effect, Case C-614/10 Commission v Austria EU:C:2012:631, paragraph 37).". The Court is holding, in other words, that an E.U. network *must* be created that can prevent user's data from being transported outside the E.U., that user's data retained for law enforcement or national defense purposes cannot be transmitted or stored outside the E.U. It will take awhile for this to be transposed into national laws. But this is very good news for folks in the E.U. and for civil libertarians globally.
Gonzalo San Gil, PhD.

Law Professor Claims Any Internet Company 'Research' On Users Without Review Board Appr... - 1 views

  •  
    "from the you-sure-you-want-to-go-there dept For many years I've been a huge fan of law professor James Grimmelmann. His legal analysis on various issues is often quite valuable, and I've quoted him more than a few times. However, he's now arguing that the now infamous Facebook happiness experiment and the similarly discussed OkCupid "hook you up with someone you should hate" experiments weren't just unethical, but illegal."
  •  
    "from the you-sure-you-want-to-go-there dept For many years I've been a huge fan of law professor James Grimmelmann. His legal analysis on various issues is often quite valuable, and I've quoted him more than a few times. However, he's now arguing that the now infamous Facebook happiness experiment and the similarly discussed OkCupid "hook you up with someone you should hate" experiments weren't just unethical, but illegal."
Gonzalo San Gil, PhD.

Free Software, Free Society - Download Here - 0 views

  •  
    "Free Software, Free Society by Richard M. Stallman Publisher: Free Software Foundation 2002 ISBN/ASIN: 1882114981 ISBN-13: 9781882114986 Number of pages: 230 Description: The intersection of ethics, law, business and computer software is the subject of these essays and speeches by MacArthur Foundation Grant winner, Richard M. Stallman. This collection includes historical writings such as The GNU Manifesto, which defined and launched the activist Free Software Movement, along with new writings on hot topics in copyright, patent law, and the controversial issue of "trusted computing." Stallman takes a critical look at common abuses of copyright law and patents when applied to computer software programs, and how these abuses damage our entire society and remove our existing freedoms."
Paul Merrell

Obama administration opts not to force firms to decrypt data - for now - The Washington... - 1 views

  • After months of deliberation, the Obama administration has made a long-awaited decision on the thorny issue of how to deal with encrypted communications: It will not — for now — call for legislation requiring companies to decode messages for law enforcement. Rather, the administration will continue trying to persuade companies that have moved to encrypt their customers’ data to create a way for the government to still peer into people’s data when needed for criminal or terrorism investigations. “The administration has decided not to seek a legislative remedy now, but it makes sense to continue the conversations with industry,” FBI Director James B. Comey said at a Senate hearing Thursday of the Homeland Security and Governmental Affairs Committee.
  • The decision, which essentially maintains the status quo, underscores the bind the administration is in — balancing competing pressures to help law enforcement and protect consumer privacy. The FBI says it is facing an increasing challenge posed by the encryption of communications of criminals, terrorists and spies. A growing number of companies have begun to offer encryption in which the only people who can read a message, for instance, are the person who sent it and the person who received it. Or, in the case of a device, only the device owner has access to the data. In such cases, the companies themselves lack “backdoors” or keys to decrypt the data for government investigators, even when served with search warrants or intercept orders.
  • The decision was made at a Cabinet meeting Oct. 1. “As the president has said, the United States will work to ensure that malicious actors can be held to account — without weakening our commitment to strong encryption,” National Security Council spokesman Mark Stroh said. “As part of those efforts, we are actively engaged with private companies to ensure they understand the public safety and national security risks that result from malicious actors’ use of their encrypted products and services.” But privacy advocates are concerned that the administration’s definition of strong encryption also could include a system in which a company holds a decryption key or can retrieve unencrypted communications from its servers for law enforcement. “The government should not erode the security of our devices or applications, pressure companies to keep and allow government access to our data, mandate implementation of vulnerabilities or backdoors into products, or have disproportionate access to the keys to private data,” said Savecrypto.org, a coalition of industry and privacy groups that has launched a campaign to petition the Obama administration.
  • ...3 more annotations...
  • To Amie Stepanovich, the U.S. policy manager for Access, one of the groups signing the petition, the status quo isn’t good enough. “It’s really crucial that even if the government is not pursuing legislation, it’s also not pursuing policies that will weaken security through other methods,” she said. The FBI and Justice Department have been talking with tech companies for months. On Thursday, Comey said the conversations have been “increasingly productive.” He added: “People have stripped out a lot of the venom.” He said the tech executives “are all people who care about the safety of America and also care about privacy and civil liberties.” Comey said the issue afflicts not just federal law enforcement but also state and local agencies investigating child kidnappings and car crashes — “cops and sheriffs . . . [who are] increasingly encountering devices they can’t open with a search warrant.”
  • One senior administration official said the administration thinks it’s making enough progress with companies that seeking legislation now is unnecessary. “We feel optimistic,” said the official, who spoke on the condition of anonymity to describe internal discussions. “We don’t think it’s a lost cause at this point.” Legislation, said Rep. Adam Schiff (D-Calif.), is not a realistic option given the current political climate. He said he made a recent trip to Silicon Valley to talk to Twitter, Facebook and Google. “They quite uniformly are opposed to any mandate or pressure — and more than that, they don’t want to be asked to come up with a solution,” Schiff said. Law enforcement officials know that legislation is a tough sell now. But, one senior official stressed, “it’s still going to be in the mix.” On the other side of the debate, technology, diplomatic and commerce agencies were pressing for an outright statement by Obama to disavow a legislative mandate on companies. But their position did not prevail.
  • Daniel Castro, vice president of the Information Technology & Innovation Foundation, said absent any new laws, either in the United States or abroad, “companies are in the driver’s seat.” He said that if another country tried to require companies to retain an ability to decrypt communications, “I suspect many tech companies would try to pull out.”
  •  
    # ! upcoming Elections...
‹ Previous 21 - 40 of 413 Next › Last »
Showing 20 items per page