Skip to main content

Home/ Open Web/ Group items tagged operational-transform

Rss Feed Group items tagged

Gary Edwards

Google Wave Operational Transformation (Google Wave Federation Protocol) - 0 views

  • Wave document operations consist of the following mutation components:skipinsert charactersinsert element startinsert element endinsert anti-element startinsert anti-element enddelete charactersdelete element startdelete element enddelete anti-element startdelete anti-element endset attributesupdate attributescommence annotationconclude annotationThe following is a more complex example document operation.skip 3insert element start with tag "p" and no attributesinsert characters "Hi there!"insert element endskip 5delete characters 4From this, one could see how an entire XML document can be represented as a single document operation. 
  • Wave OperationsWave operations consists of a document operation, for modifying XML documents and other non document operations. Non document operations are for tasks such as adding or removing a participant to a Wavelet. We'll focus on document operations here as they are the most central to Wave.It's worth noting that an XML document in Wave can be regarded as a single document operation that can be applied to the empty document.This section will also cover how Wave operations are particularly efficient even in the face of a large number of transforms.XML Document SupportWave uses a streaming interface for document operations. This is similar to an XMLStreamWriter or a SAX handler. The document operation consists of a sequence of ordered document mutations. The mutations are applied in sequence as you traverse the document linearly. Designing document operations in this manner makes it easier to write transformation function and composition function described later.In Wave, every 16-bit Unicode code unit (as used in javascript, JSON, and Java strings), start tag or end tag in an XML document is called an item. Gaps between items are called positions. Position 0 is before the first item. A document operation can contain mutations that reference positions. For example, a "Skip" mutation specifies how many positions to skip ahead in the XML document before applying the next mutation.Wave document operations also support annotations. An annotation is some meta-data associated with an item range, i.e. a start position and an end position. This is particularly useful for describing text formatting and spelling suggestions, as it does not unecessarily complicate the underlying XML document format.
  •  
    Summary: Collaborative document editing means multiple editors being able to edit a shared document at the same time.. Live and concurrent means being able to see the changes another person is making, keystroke by keystroke. Currently, there are already a number of products on the market that offer collaborative document editing. Some offer live concurrent editing, such as EtherPad and SubEthaEdit, but do not offer rich text. There are others that offer rich text, such as Google Docs, but do not offer a seamless live concurrent editing experience, as merge failures can occur. Wave stands as a solution that offers both live concurrent editing and rich text document support.  The result is that Wave allows for a very engaging conversation where you can see what the other person is typing, character by character much like how you would converse in a cafe. This is very much like instant messaging except you can see what the other person is typing, live. Wave also allows for a more productive collaborative document editing experience, where people don't have to worry about stepping on each others toes and still use common word processor functionalities such as bold, italics, bullet points, and headings. Wave is more than just rich text documents. In fact, Wave's core technology allows live concurrent modifications of XML documents which can be used to represent any structured content including system data that is shared between clients and backend systems. To achieve these goals, Wave uses a concurrency control system based on Operational Transformation.
Paul Merrell

InfoQ: Google Wave's Architecture - 0 views

  • Operational Transformation This is the crucial part of Wave’s technology. Google Wave makes extensive use of Operational Transformations (OT) which are executed on the server. When an user edits a collaborative document opened by several users, the client program provides an Optimistic UI by immediately displaying what he/she types but it also sends the editing operation to the server to be ratified hoping that it will be accepted by the server. The client waits for the server to evaluate the operation and will cache any other operations until the server replies. After the server replies, all cached operations are sent from client to server in bulk. The server, considering operations received from other clients, will transform the operation accordingly and will inform all clients about the transformation, and the clients will update their UI accordingly. Operations are sent to the server and propagated to each client on a character by character basis, unless it is a bulk operation. The server is the keeper of the document and its version is considered the “correct” version. In the end, each client will be updated with the final version received from the server, which is the result of possibly many operational transformations. There are recovery means provided for communication failure or server/client crash. All XML documents exchanged between the client and the server carry a checksum for rapid identification of miscommunications.
Gary Edwards

XML Production Workflows? Start with the Web and XHTML - 1 views

  • Challenges: Some Ugly Truths The challenges of building—and living with—an XML workflow are clear enough. The return on investment is a long-term proposition. Regardless of the benefits XML may provide, the starting reality is that it represents a very different way of doing things than the one we are familiar with. The Word Processing and Desktop Publishing paradigm, based on the promise of onscreen, WYSIWYG layout, is so dominant as to be practically inescapable. It has proven really hard to get from here to there, no matter how attractive XML might be on paper. A considerable amount of organizational effort and labour must be expended up front in order to realize the benefits. This is why XML is often referred to as an “investment”: you sink a bunch of time and money up front, and realize the benefits—greater flexibility, multiple output options, searching and indexing, and general futureproofing—later, over the long haul. It is not a short-term return proposition. And, of course, the returns you are able to realize from your XML investment are commensurate with what you put in up front: fine-grained, semantically rich tagging is going to give you more potential for searchability and recombination than a looser, more general-purpose approach, but it sure costs more. For instance, the Text Encoding Initiative (TEI) is the grand example of pouring enormous amounts of energy into the up-front tagging, with a very open-ended set of possibilities down the line. TEI helpfully defines a level to which most of us do not have to aspire.[5] But understanding this on a theoretical level is only part of the challenge. There are many practical issues that must be addressed. Software and labour are two of the most critical. How do you get the content into XML in the first place? Unfortunately, despite two decades of people doing SGML and XML, this remains an ugly question.
  • Practical Challenges In 2009, there is still no truly likeable—let alone standard—editing and authoring software for XML. For many (myself included), the high-water mark here was Adobe’s FrameMaker, substantially developed by the late 1990s. With no substantial market for it, it is relegated today mostly to the tech writing industry, unavailable for the Mac, and just far enough afield from the kinds of tools we use today that its adoption represents a significant hurdle. And FrameMaker was the best of the breed; most of the other software in decent circulation are programmers’ tools—the sort of things that, as Michael Tamblyn pointed out, encourage editors to drink at their desks. The labour question represents a stumbling block as well. The skill-sets and mind-sets that effective XML editors need have limited overlap with those needed by literary and more traditional production editors. The need to think of documents as machine-readable databases is not something that comes naturally to folks steeped in literary culture. In combination with the sheer time and effort that rich tagging requires, many publishers simply outsource the tagging to India, drawing a division of labour that spans oceans, to put it mildly. Once you have XML content, then what do you do with it? How do you produce books from it? Presumably, you need to be able to produce print output as well as digital formats. But while the latter are new enough to be generally XML-friendly (e-book formats being largely XML based, for instance), there aren’t any straightforward, standard ways of moving XML content into the kind of print production environments we are used to seeing. This isn’t to say that there aren’t ways of getting print—even very high-quality print—output from XML, just that most of them involve replacing your prepress staff with Java programmers.
  • Why does this have to be so hard? It’s not that XML is new, or immature, or untested. Remember that the basics have been around, and in production, since the early 1980s at least. But we have to take account of a substantial and long-running cultural disconnect between traditional editorial and production processes (the ones most of us know intimately) and the ways computing people have approached things. Interestingly, this cultural divide looked rather different in the 1970s, when publishers were looking at how to move to digital typesetting. Back then, printers and software developers could speak the same language. But that was before the ascendancy of the Desktop Publishing paradigm, which computerized the publishing industry while at the same time isolating it culturally. Those of us who learned how to do things the Quark way or the Adobe way had little in common with people who programmed databases or document-management systems. Desktop publishing technology isolated us in a smooth, self-contained universe of toolbars, grid lines, and laser proofs. So, now that the reasons to get with this program, XML, loom large, how can we bridge this long-standing divide?
  • ...44 more annotations...
  • Using the Web as a Production Platform The answer, I think, is right in front of you. The bridge is the Web, a technology and platform that is fundamentally based on XML, and which many publishers are by now comfortably familiar with. Perhaps not entirely comfortably, but at least most publishers are already working with the Web; they already either know or have on staff people who understand it and can work with it. The foundation of our argument is this: rather than looking at jumping to XML in its full, industrial complexity, which seems to be what the O'Reilly-backed StartWithXML initiative[6] is suggesting, publishers instead leverage existing tools and technologies—starting with the Web—as a means of getting XML workflows in place. This means making small investments and working with known tools rather than spending tens of thousands of dollars on XML software and rarefied consultants. It means re-thinking how the existing pieces of the production toolchain fit together; re-thinking the existing roles of software components already in use. It means, fundamentally, taking the Web seriously as a content platform, rather than thinking of it as something you need to get content out to, somehow. If nothing else, the Web represents an opportunity to think about editorial and production from outside the shrink-wrapped Desktop Publishing paradigm.
  • Is the Web made of Real XML? At this point some predictable objections can be heard: wait a moment, the Web isn’t really made out of XML; the HTML that makes up most of the Web is at best the bastard child of SGML, and it is far too flaky/unstructured/underpowered to be taken seriously. We counter by arguing that although HTML on the Web exists in a staggering array of different incarnations, and that the majority of it is indeed an unstructured mess, this does not undermine the general principle that basic, ubiquitous Web technologies can make a solid platform for content management, editorial process, and production workflow.
  • With the advent of a published XML standard in the late 1990s came the W3C’s adoption of XHTML: the realization of the Web’s native content markup as a proper XML document type. Today, its acceptance is almost ubiquitous, even while the majority of actual content out there may not be strictly conforming. The more important point is that most contemporary Web software, from browsers to authoring tools to content management systems (from blogs to enterprise systems), are capable of working with clean, valid XHTML. Or, to put the argument the other way around, clean, valid XHTML content plays absolutely seamlessly with everything else on the Web.[7]
  • The objection which follows, then, will be that even if we grant that XHTML is a real XML document type, that it is underpowered for “serious” content because it is almost entirely presentation (formatting) oriented; it lacks any semantic depth. In XHTML, a paragraph is a paragraph is a paragraph, as opposed to a section or an epigraph or a summary.
  • n contrast, more “serious” XML document types like DocBook[8] or DITA-derived schemas[9] are capable of making semantic distinctions about content chunks at a fine level of granularity and with a high degree of specificity.
  • So there is an argument for recalling the 80:20 rule here. If XHTML can provide 80% of the value with just 20% of the investment, then what exactly is the business case for spending the other 80% to achieve that last 20% of value? We suspect the ratio is actually quite a bit steeper than 80:20 for most publishers.
  • Furthermore, just to get technical for a moment, XHTML is extensible in a fairly straightforward way, through the common “class” attribute on each element. Web developers have long leveraged this kind of extensibility in the elaboration of “microformats” for semantic-web applications.[10] There is no reason why publishers shouldn’t think to use XHTML’s simple extensibility in a similar way for their own ends.
  • XHTML, on the other hand, is supported by a vast array of quotidian software, starting with the ubiquitous Web browser. For this very reason, XHTML is in fact employed as a component part of several more specialized document types (ONIX and ePub among them).
  • Why re-invent a general-purpose prose representation when XHTML already does the job?
  • It is worth pausing for a moment to consider the role of XHTML in the ePub standard for ebook content. An ePub file is, anatomically, a simply disguised zip archive. Inside the zip archive are a few standard component parts: there are specialized files that declare metadata about the book, and about the format of the book. And then there is the book’s content, represented in XHTML. An ePub book is a Web page in a wrapper.
  • To sum up the general argument: the Web as it already exists presents incredible value to publishers, as a platform for doing XML content management with existing (and often free) tools, and without having to go blindly into the unknown. At this point, we can offer a few design guidelines: prefer existing and/or ubiquitous tools over specialized ones wherever possible; prefer free software over proprietary systems where possible; prefer simple tools controlled and coordinated by human beings over fully automated (and therefore complex) systems; play to our strengths: use Web software for storing and managing content, use layout software for layout, and keep editors and production people in charge of their own domains.
  • Putting the Pieces Together: A Prototype
  • At the SFU Master of Publishing Program, we have been chipping away at this general line of thinking for a few years. Over that time, Web content management systems have been getting more and more sophisticated, all the while getting more streamlined and easier to use. (NB: if you have a blog, you have a Web content management system.) The Web is beginning to be recognized as a writing and editing environment used by millions of people. And the ways in which content is represented, stored, and exchanged online have become increasingly robust and standardized.
  • The missing piece of the puzzle has been print production: how can we move content from its malleable, fluid form on line into the kind of high-quality print production environments we’ve come to expect after two decades of Desktop Publishing?
  • Anyone who has tried to print Web content knows that the existing methods leave much to be desired (hyphenation and justification, for starters). In the absence of decent tools for this, most publishers quite naturally think of producing the print content first, and then think about how to get material onto the Web for various purposes. So we tend to export from Word, or from Adobe, as something of an afterthought.
  • While this sort of works, it isn’t elegant, and it completely ignores the considerable advantages of Web-based content management.
  • Content managed online is stored in one central location, accessible simultaneously to everyone in your firm, available anywhere you have an Internet connection, and usually exists in a much more fluid format than Word files. If only we could manage the editorial flow online, and then go to print formats at the end, instead of the other way around. At SFU, we made several attempts to make this work by way of the supposed “XML import” capabilities of various Desktop Publishing tools, without much success.[12]
  • In the winter of 2009, Adobe solved this part of the problem for us with the introduction of its Creative Suite 4. What CS4 offers is the option of a complete XML representation of an InDesign document: what Adobe calls IDML (InDesign Markup Language).
  • The IDML file format is—like ePub—a simply disguised zip archive that, when unpacked, reveals a cluster of XML files that represent all the different facets of an InDesign document: layout spreads, master pages, defined styles, colours, and of course, the content.
  • IDML is a well thought-out XML standard that achieves two very different goals simultaneously: it preserves all of the information that InDesign needs to do what it does; and it is broken up in a way that makes it possible for mere mortals (or at least our Master of Publishing students) to work with it.
  • What this represented to us in concrete terms was the ability to take Web-based content and move it into InDesign in a straightforward way, thus bridging Web and print production environments using existing tools and skillsets, with a little added help from free software.
  • We would take clean XHTML content, transform it to IDML-marked content, and merge that with nicely designed templates in InDesign.
  • The result is an almost push-button publication workflow, which results in a nice, familiar InDesign document that fits straight into the way publishers actually do production.
  • Tracing the steps To begin with, we worked backwards, moving the book content back to clean XHTML.
  • The simplest method for this conversion—and if you want to create Web content, this is an excellent route—was to use Adobe’s “Export to Digital Editions” option, which creates an ePub file.
  • Recall that ePub is just XHTML in a wrapper, so within the ePub file was a relatively clean XHTML document. It was somewhat cleaner (that is, the XHTML tagging was simpler and less cluttered) than InDesign’s other Web-oriented exports, possibly because Digital Editions is a well understood target, compared with somebody’s website.
  • In order to achieve our target of clean XHTML, we needed to do some editing; the XHTML produced by InDesign’s “Digital Editions” export was presentation-oriented. For instance, bulleted list items were tagged as paragraphs, with a class attribute identifying them as list items. Using the search-and-replace function, we converted such structures to proper XHTML list and list-item elements. Our guiding principle was to make the XHTML as straightforward as possible, not dependent on any particular software to interpret it.
  • We broke the book’s content into individual chapter files; each chapter could then carry its own basic metadata, and the pages conveniently fit our Web content management system (which is actually just a wiki). We assembled a dynamically generated table of contents for the 12 chapters, and created a cover page. Essentially, the book was entirely Web-based at this point.
  • When the book chapters are viewed online, they are formatted via a CSS2 stylesheet that defines a main column for content as well as dedicating screen real estate for navigational elements. We then created a second template to render the content for exporting; this was essentially a bare-bones version of the book with no navigation and minimal styling. Pages (or even the entire book) can be exported (via the “Save As...” function in a Web browser) for use in either print production or ebook conversion. At this point, we required no skills beyond those of any decent Web designer.
  • Integrating with CS4 for Print Adobe’s IDML language defines elements specific to InDesign; there is nothing in the language that looks remotely like XHTML. So a mechanical transformation step is needed to convert the XHTML content into something InDesign can use. This is not as hard as it might seem.
  • Both XHTML and IDML are composed of straightforward, well-documented structures, and so transformation from one to the other is, as they say, “trivial.” We chose to use XSLT (Extensible Stylesheet Language Transforms) to do the work. XSLT is part of the overall XML specification, and thus is very well supported in a wide variety of tools. Our prototype used a scripting engine called xsltproc, a nearly ubiquitous piece of software that we found already installed as part of Mac OS X (contemporary Linux distributions also have this as a standard tool), though any XSLT processor would work.
  • In other words, we don’t need to buy InCopy, because we just replaced it with the Web. Our wiki is now plugged directly into our InDesign layout. It even automatically updates the InDesign document when the content changes. Credit is due at this point to Adobe: this integration is possible because of the open file format in the Creative Suite 4.
  • We wrote an XSLT transformation script[18] that converted the XHTML content from the Web into an InCopy ICML file. The script itself is less than 500 lines long, and was written and debugged over a period of about a week by amateurs (again, the people named at the start of this article). The script runs in a couple of seconds, and the resulting .icml file can then be “placed” directly into an InDesign template. The ICML file references an InDesign stylesheet, so the template file can be set up with a house-styled layout, master pages, and stylesheet definitions for paragraphs and character ranges.
  • Rather than a public-facing website, our system relies on the Web as a content management platform—of course a public face could easily be added.
  • It should be noted that the Book Publishing 1 proof-of-concept was artificially complex; we began with a book laid out in InDesign and ended up with a look-alike book laid out in InDesign. But next time—for instance, when we publish Book Publishing 2—we can begin the process with the content on the Web, and keep it there throughout the editorial process. The book’s content could potentially be written and edited entirely online, as Web content, and then automatically poured into an InDesign template at proof time. “Just in time,” as they say. This represents an entirely new way of thinking of book production. With a Web-first orientation, it makes little sense to think of the book as “in print” or “out of print”—the book is simply available, in the first place online; in the second place in derivative digital formats; and third, but really not much more difficult, in print-ready format, via the usual InDesign CS print production system publishers are already familiar with.
  • Creating Ebook Files Creating electronic versions from XHTML source is vastly simpler than trying to generate these out of the existing print process. The ePub version is extremely easy to generate; so is online marketing copy or excerpts for the Web, since the content begins life Web-native.
  • Since an ePub file is essentially XHTML content in a special wrapper, all that is required is that we properly “wrap” our XHTML content. Ideally, the content in an ePub file is broken into chapters (as ours was) and a table of contents file is generated in order to allow easy navigation within an ebook reader. We used Julian Smart’s free tool eCub[19] to simply and automatically generate the ePub wrapper and the table of contents. The only custom development we did was to create a CSS stylesheet for the ebook so that headings and paragraph indents looked the way we wanted. Starting with XHTML content, creating ePub is almost too easy.
  • today, we are able to put the process together using nothing but standard, relatively ubiquitous Web tools: the Web itself as an editing and content management environment, standard Web scripting tools for the conversion process, and the well-documented IDML file format to integrate the layout tool.
  • Our project demonstrates that Web technologies are indeed good enough to use in an XML-oriented workflow; more specialized and expensive options are not necessarily required. For massive-scale enterprise publishing, this approach may not offer enough flexibility, and the challenge of adding and extracting extra semantic richness may prove more trouble than it's worth.
  • But for smaller firms who are looking at the straightforward benefits of XML-based processes—single source publishing, online content and workflow management, open and accessible archive formats, greater online discoverability—here is a way forward.
  • The result is very simple and easy to use. Our demonstration requires that a production editor run the XSLT transformation script manually, but there is no reason why this couldn’t be built directly into the Web content management system so that exporting the content to print ran the transformation automatically. The resulting file would then be “placed” in InDesign and proofed.
  • The final piece of our puzzle, the ability to integrate print production, was made possible by Adobe's release of InDesign with an open XML file format. Since the Web's XHTML is also XML, is can be easily and confidently transformed to the InDesign format.
  • Such a workflow—beginning with the Web and exporting to print—is surely more in line with the way we will do business in the 21st century, where the Web is the default platform for reaching audiences, developing content, and putting the pieces together. It is time, we suggest, for publishers to re-orient their operations and start with the Web.
  • Using the Web as a Production Platform
  •  
    I was looking for an answer to a problem Marbux had presented, and found this interesting article.  The issue was that of the upcoming conversion of the Note Case Pro (NCP) layout engine to the WebKit layout engine, and what to do about the NCP document format. My initial reaction was to encode the legacy NCP document format in XML, and run an XSLT to a universal pivot format like TEI-XML.  From there, the TEI-XML community would provide all the XSLT transformation routines for conversion to ODF, OOXML, XHTML, ePUB and HTML/CSS. Researching the problems one might encounter with this approach, I found this article.  Fascinating stuff. My take away is that TEI-XML would not be as effective a "universal pivot point" as XHTML.  Or perhaps, if NCP really wants to get aggressive; IDML - InDesign Markup Language. As an after thought, i was thinking that an alternative title to this article might have been, "Working with Web as the Center of Everything".
Gary Edwards

Google Open Sources Heart and Soul of Google Wave Code - 0 views

  •  
    Google programmers open source two components of the Google Wave messaging and collaboration prototype. One includes the Operational Transform, which forms the complex center of the Wave model. Google Wave is an example of the Pushbutton Web, where real-time communications rule the roost. Google July 24 said it released to open source the OT (Operational Transform) code, the framework that enables multiple people to edit a single document in real time across a wide-area network, as well as a basic client/server prototype that uses the wave protocol. The Google Wave Federation Protocol is an open extension to the XMPP core protocol, geared to allow near real-time communication of wave updates between two wave servers.
Gary Edwards

That's All Folks: Why the Writing Is on the Wall at Microsoft - Forbes - 1 views

  •  
    Control vs. Creativity.  As the ulitmate control freak, Ballmer was guanteed to crush the life out of Microsoft's most creative individuals and teams.  Gates was focused on Windows and MSOffice, and Ballmer on control.  That one-two punch made certain that Microsoft would not be a player in the next great wave of computing; The Cloud.   excerpt: This is yet another example of what I like to call the Wile E. Coyote syndrome. Like the unfortunate character in the old Warner Bros. cartoons, Microsoft now seems to be a company that has long since run off the cliff but, with legs spinning for all they are worth, doesn't know yet that it is ready to drop. Yet drop it most certainly will. Microsoft Win8 Tablet Is NOT a Game Changer Adam Hartung Contributor Snapshot: Steve Ballmer Follow (93) #44 Billionaires IDC Analyst: Microsoft's Surface Sizzle Needs Win8 Steak Daniel Nye Griffiths Contributor To understand how this happens, take a look at the work of Arnold J. Toynbee, a historian who studied the rise and fall of civilizations. He argued that a civilization flourishes when it motivates insiders and attracts outsiders with its creative dynamism and culture. The civilization breaks down when its leadership loses this creative capacity and gives way to, or transforms itself into, a dominant minority. When this happens, the driver of the civilization becomes control, not attraction. And it's precisely this switch from attraction to control that is the source of the breakdown. Interestingly, Toynbee says that the consequences may not be immediately apparent. A civilization can keep up momentum because the controls it puts in place generate some short-term efficiency. But eventually it will run its course and collapse, because no amount of control can replace the loss of collective creativity. Observe this in the corporate world by looking at the example of General Motors. G.M.'s 2009 bankruptcy came at the end of a long decline dating back to the early 197
Paul Merrell

The People and Tech Behind the Panama Papers - Features - Source: An OpenNews project - 0 views

  • Then we put the data up, but the problem with Solr was it didn’t have a user interface, so we used Project Blacklight, which is open source software normally used by librarians. We used it for the journalists. It’s simple because it allows you to do faceted search—so, for example, you can facet by the folder structure of the leak, by years, by type of file. There were more complex things—it supports queries in regular expressions, so the more advanced users were able to search for documents with a certain pattern of numbers that, for example, passports use. You could also preview and download the documents. ICIJ open-sourced the code of our document processing chain, created by our web developer Matthew Caruana Galizia. We also developed a batch-searching feature. So say you were looking for politicians in your country—you just run it through the system, and you upload your list to Blacklight and you would get a CSV back saying yes, there are matches for these names—not only exact matches, but also matches based on proximity. So you would say “I want Mar Cabra proximity 2” and that would give you “Mar Cabra,” “Mar whatever Cabra,” “Cabra, Mar,”—so that was good, because very quickly journalists were able to see… I have this list of politicians and they are in the data!
  • Last Sunday, April 3, the first stories emerging from the leaked dataset known as the Panama Papers were published by a global partnership of news organizations working in coordination with the International Consortium of Investigative Journalists, or ICIJ. As we begin the second week of reporting on the leak, Iceland’s Prime Minister has been forced to resign, Germany has announced plans to end anonymous corporate ownership, governments around the world launched investigations into wealthy citizens’ participation in tax havens, the Russian government announced that the investigation was an anti-Putin propaganda operation, and the Chinese government banned mentions of the leak in Chinese media. As the ICIJ-led consortium prepares for its second major wave of reporting on the Panama Papers, we spoke with Mar Cabra, editor of ICIJ’s Data & Research unit and lead coordinator of the data analysis and infrastructure work behind the leak. In our conversation, Cabra reveals ICIJ’s years-long effort to build a series of secure communication and analysis platforms in support of genuinely global investigative reporting collaborations.
  • For communication, we have the Global I-Hub, which is a platform based on open source software called Oxwall. Oxwall is a social network, like Facebook, which has a wall when you log in with the latest in your network—it has forum topics, links, you can share files, and you can chat with people in real time.
  • ...3 more annotations...
  • We had the data in a relational database format in SQL, and thanks to ETL (Extract, Transform, and Load) software Talend, we were able to easily transform the data from SQL to Neo4j (the graph-database format we used). Once the data was transformed, it was just a matter of plugging it into Linkurious, and in a couple of minutes, you have it visualized—in a networked way, so anyone can log in from anywhere in the world. That was another reason we really liked Linkurious and Neo4j—they’re very quick when representing graph data, and the visualizations were easy to understand for everybody. The not-very-tech-savvy reporter could expand the docs like magic, and more technically expert reporters and programmers could use the Neo4j query language, Cypher, to do more complex queries, like show me everybody within two degrees of separation of this person, or show me all the connected dots…
  • We believe in open source technology and try to use it as much as possible. We used Apache Solr for the indexing and Apache Tika for document processing, and it’s great because it processes dozens of different formats and it’s very powerful. Tika interacts with Tesseract, so we did the OCRing on Tesseract. To OCR the images, we created an army of 30–40 temporary servers in Amazon that allowed us to process the documents in parallel and do parallel OCR-ing. If it was very slow, we’d increase the number of servers—if it was going fine, we would decrease because of course those servers have a cost.
  • For the visualization of the Mossack Fonseca internal database, we worked with another tool called Linkurious. It’s not open source, it’s licensed software, but we have an agreement with them, and they allowed us to work with it. It allows you to represent data in graphs. We had a version of Linkurious on our servers, so no one else had the data. It was pretty intuitive—journalists had to click on dots that expanded, basically, and could search the names.
Paul Merrell

Microsoft to host data in Germany to evade US spying | Naked Security - 0 views

  • Microsoft's new plan to keep the US government's hands off its customers' data: Germany will be a safe harbor in the digital privacy storm. Microsoft on Wednesday announced that beginning in the second half of 2016, it will give foreign customers the option of keeping data in new European facilities that, at least in theory, should shield customers from US government surveillance. It will cost more, according to the Financial Times, though pricing details weren't forthcoming. Microsoft Cloud - including Azure, Office 365 and Dynamics CRM Online - will be hosted from new datacenters in the German regions of Magdeburg and Frankfurt am Main. Access to data will be controlled by what the company called a German data trustee: T-Systems, a subsidiary of the independent German company Deutsche Telekom. Without the permission of Deutsche Telekom or customers, Microsoft won't be able to get its hands on the data. If it does get permission, the trustee will still control and oversee Microsoft's access.
  • Microsoft CEO Satya Nadella dropped the word "trust" into the company's statement: Microsoft’s mission is to empower every person and every individual on the planet to achieve more. Our new datacenter regions in Germany, operated in partnership with Deutsche Telekom, will not only spur local innovation and growth, but offer customers choice and trust in how their data is handled and where it is stored.
  • On Tuesday, at the Future Decoded conference in London, Nadella also announced that Microsoft would, for the first time, be opening two UK datacenters next year. The company's also expanding its existing operations in Ireland and the Netherlands. Officially, none of this has anything to do with the long-drawn-out squabbling over the transatlantic Safe Harbor agreement, which the EU's highest court struck down last month, calling the agreement "invalid" because it didn't protect data from US surveillance. No, Nadella said, the new datacenters and expansions are all about giving local businesses and organizations "transformative technology they need to seize new global growth." But as Diginomica reports, Microsoft EVP of Cloud and Enterprise Scott Guthrie followed up his boss’s comments by saying that yes, the driver behind the new datacenters is to let customers keep data close: We can guarantee customers that their data will always stay in the UK. Being able to very concretely tell that story is something that I think will accelerate cloud adoption further in the UK.
  • ...2 more annotations...
  • Microsoft and T-Systems' lawyers may well think that storing customer data in a German trustee data center will protect it from the reach of US law, but for all we know, that could be wishful thinking. Forrester cloud computing analyst Paul Miller: To be sure, we must wait for the first legal challenge. And the appeal. And the counter-appeal. As with all new legal approaches, we don’t know it is watertight until it is challenged in court. Microsoft and T-Systems’ lawyers are very good and say it's watertight. But we can be sure opposition lawyers will look for all the holes. By keeping data offshore - particularly in Germany, which has strong data privacy laws - Microsoft could avoid the situation it's now facing with the US demanding access to customer emails stored on a Microsoft server in Dublin. The US has argued that Microsoft, as a US company, comes under US jurisdiction, regardless of where it keeps its data.
  • Running away to Germany isn't a groundbreaking move; other US cloud services providers have already pledged expansion of their EU presences, including Amazon's plan to open a UK datacenter in late 2016 that will offer what CTO Werner Vogels calls "strong data sovereignty to local users." Other big data operators that have followed suit: Salesforce, which has already opened datacenters in the UK and Germany and plans to open one in France next year, as well as new EU operations pledged for the new year by NetSuite and Box. Can Germany keep the US out of its datacenters? Can Ireland? Time, and court cases, will tell.
  •  
    The European Community's Court of Justice decision in the Safe Harbor case --- and Edward Snowden --- are now officially downgrading the U.S. as a cloud data center location. NSA is good business for Europeans looking to displace American cloud service providers, as evidenced by Microsoft's decision. The legal test is whether Microsoft has "possession, custody, or control" of the data. From the info given in the article, it seems that Microsoft has done its best to dodge that bullet by moving data centers to Germany and placing their data under the control of a European company. Do ownership of the hardware and profits from their rent mean that Microsoft still has "possession, custody, or control" of the data? The fine print of the agreement with Deutsche Telekom and the customer EULAs will get a thorough going over by the Dept. of Justice for evidence of Microsoft "control" of the data. That will be the crucial legal issue. The data centers in Germany may pass the test. But the notion that data centers in the UK can offer privacy is laughable; the UK's legal authority for GCHQ makes it even easier to get the data than the NSA can in the U.S.  It doesn't even require a court order. 
Paul Merrell

Google Releases Realtime API For Drive Apps - Development - Mobility - 0 views

  • Google has released a new application programming interface (API) that allows developers to implement real-time collaboration in Google Drive apps. Users of Google Docs, as well as Spreadsheets and Slides, now have the ability to edit a document at the same time others are doing so, and each can see the changes input by collaborators in real time. This is made possible by a technology called operational transformation, also featured in the now-discontinued Google Wave, which ensures the rapid transference of changes over a network.
  • Now developers who create apps that rely on Google Drive for storage can provide their users with the ability to interact and work together in real time. "With the new Google Drive Realtime API, you can now easily add some of the same real-time collaboration that powers Google Drive to your own apps," explained Brian Cairns, a software engineer at Google, in a blog post. "This new API handles network communication, storage, presence, conflict resolution and other collaborative details so you can focus on building great apps."
Gary Edwards

Cloud computing, virtualisation top Gartner CIO survey - 0 views

  • It is these constrained budgets that will drive enterprise adoption of cloud services and virtualisation, McDonald said."These technologies were selected by CIOs the most often and are the top-two technologies for 2011, and are well-suited for this budget reality," he commented. "They offer similar service levels at lower budget costs."
  • rise to 43% over the next four years
  •  
    Cloud computing and virtualisation are the top two technology priorities for CIOs in 2011, according to the results of a survey published on Friday by Gartner that revealed global IT budgets are likely to remain largely flat this year. Networking, voice and data communications - traditionally the domain of telcos - ranks sixth in the research firm's study. "New lighter-weight technologies - such as cloud computing, software as a service (SaaS), and social networks - and IT models enable the CIO to redefine IT, giving it a greater focus on growth and strategic impact," said a statement from Mark McDonald, group vice president and head of research for Gartner Executive Programs (EXP). Indeed, Gartner's survey also found that CIOs expect Internet service-based technologies will allow them to divert more resources - up to 50% of their budgets - away from day-to-day operations and towards transforming their business strategies, which could prove significant in the wake of the recession.
Gary Edwards

Why Cloud Computing is the Future of Mobile - 0 views

  •  
    This one's for Florian. He's been wondering about mobile computing and that creeping sense of being left out of something big. The desktop is so not happening. It's day has come and gone. Now there is a study out from ABI Research, connecting mobile computing to the future of the Web. Good stuff: Intro Excerpt:The term "cloud computing" is being bandied about a lot these days, mainly in the context of the "future of the web." But cloud computing's potential doesn't begin and end with the personal computer's transformation into a thin client - the mobile platform is going to be heavily impacted by this technology as well. At least that's the analysis being put forth by ABI Research. Their recent report, Mobile Cloud Computing, theorizes that the cloud will soon become a disruptive force in the mobile world, eventually becoming the dominant way in which mobile applications operate.
Gary Edwards

Changing technology - How cloud computing is transforming business - and why you should... - 0 views

  •  
    If someone told you that you could drop your operating costs by 40 percent, would you listen? If that same person said you could save between $70 and $150 per user per year in energy savings alone if you tried something new, would you try it?  A lot of companies are listening, and those same businesses are trying something new - cloud computing and software as a service (SaaS) - and reaping the many benefits, which start with the aforementioned cost savings. "It's about saving money, and there's a tremendous amount of money to be saved, because if you look at IT budgets, nearly 80 percent of that budget, in many cases, is spent just to keep the lights on, which means the other 20 percent is the only money that's actually able to be used to implement new technologies into the model," says Jeff McNaught, chief marketing officer at Wyse Technology Inc. 
Paul Merrell

New York City to turn phone booths into Wi-Fi hot spots | Al Jazeera America - 0 views

  • New York City Mayor Bill de Blasio is fielding proposals to transform the city’s largely forgotten phone booths into Wi-Fi hot spots, an ambitious project that would create one of the largest public Wi-Fi networks in the country. The team with the winning proposal will be charged with the installation, operation and maintenance of up to 10,000 hot spots distributed across the five boroughs, according to a statement released Thursday by the mayor’s office.
Paul Merrell

Ericsson Sues to Block Apple iPhone in U.S. Amid Patent Spat | Bloomberg BNA - 0 views

  • The licensing battle between Apple Inc. and Ericsson AB is escalating.Ericsson, a pioneer in mobile phones that transformed itself into the world's largest maker of wireless networks, said Friday it's filing seven new lawsuits in a U.S. court and is asking the U.S. International Trade Commission to block Apple products from the U.S. market.Together, the complaints accuse Apple of infringing as many as 41 patents for some of the fundamental ways mobile devices communicate and for related technology such as user interfaces, battery saving and the operating system.
  • Apple had been paying royalties to Stockholm-based Ericsson before a license expired in mid-January. When talks over renewal failed, the companies sued each other, seeking court rulings on whether Ericsson's royalty demands on fundamental technology were fair and reasonable.
  • The new complaints being filed by Ericsson at the International Trade Commission in Washington take the dispute to another level and are designed to put pressure on Apple. The trade commission, whose job is to protect U.S. markets from unfair trade practices, moves more swiftly than district courts and has the power to block products from crossing the border.Apple's iPhone, iPad and other devices are made in Asia.
Paul Merrell

India begins to embrace digital privacy. - 0 views

  • India is the world’s largest democracy and is home to 13.5 percent of the world’s internet users. So the Indian Supreme Court’s August ruling that privacy is a fundamental, constitutional right for all of the country’s 1.32 billion citizens was momentous. But now, close to three months later, it’s still unclear exactly how the decision will be implemented. Will it change everything for internet users? Or will the status quo remain? The most immediate consequence of the ruling is that tech companies such as Facebook, Twitter, Google, and Alibaba will be required to rein in their collection, utilization, and sharing of Indian user data. But the changes could go well beyond technology. If implemented properly, the decision could affect national politics, business, free speech, and society. It could encourage the country to continue to make large strides toward increased corporate and governmental transparency, stronger consumer confidence, and the establishment and growth of the Indian “individual” as opposed to the Indian collective identity. But that’s a pretty big if. Advertisement The privacy debate in India was in many ways sparked by a controversy that has shaken up the landscape of national politics for several months. It began in 2016 as a debate around a social security program that requires participating citizens to obtain biometric, or Aadhaar, cards. Each card has a unique 12-digit number and records an individual’s fingerprints and irises in order to confirm his or her identity. The program was devised to increase the ease with which citizens could receive social benefits and avoid instances of fraud. Over time, Aadhaar cards have become mandatory for integral tasks such as opening bank accounts, buying and selling property, and filing tax returns, much to the chagrin of citizens who are uncomfortable about handing over their personal data. Before the ruling, India had weak privacy protections in place, enabling unchecked data collection on citizens by private companies and the government. Over the past year, a number of large-scale data leaks and breaches that have impacted major Indian corporations, as well as the Aadhaar program itself, have prompted users to start asking questions about the security and uses of their personal data.
  • n order to bolster the ruling the government will also be introducing a set of data protection laws that are to be developed by a committee led by retired Supreme Court judge B.N. Srikrishna. The committee will study the data protection landscape, develop a draft Data Protection Bill, and identify how, and whether, the Aadhaar Act should be amended based on the privacy ruling.
  • Should the data protection laws be implemented in an enforceable manner, the ruling will significantly impact the business landscape in India. Since the election of Prime Minister Narendra Modi in May 2014, the government has made fostering and expanding the technology and startup sector a top priority. The startup scene has grown, giving rise to several promising e-commerce companies, but in 2014, only 12 percent of India’s internet users were online consumers. If the new data protection laws are truly impactful, companies will have to accept responsibility for collecting, utilizing, and protecting user data safely and fairly. Users would also have a stronger form of redress when their newly recognized rights are violated, which could transform how they engage with technology. This has the potential to not only increase consumer confidence but revitalize the Indian business sector, as it makes it more amenable and friendly to outside investors, users, and collaborators.
Paul Merrell

Google will 'de-rank' RT articles to make them harder to find - Eric Schmidt - RT World... - 0 views

  • Eric Schmidt, the Executive Chairman of Google’s parent company Alphabet, says the company will “engineer” specific algorithms for RT and Sputnik to make their articles less prominent on the search engine’s news delivery services. “We are working on detecting and de-ranking those kinds of sites – it’s basically RT and Sputnik,” Schmidt said during a Q & A session at the Halifax International Security Forum in Canada on Saturday, when asked about whether Google facilitates “Russian propaganda.”
  • “We are well of aware of it, and we are trying to engineer the systems to prevent that [the content being delivered to wide audiences]. But we don’t want to ban the sites – that’s not how we operate.”The discussion focused on the company’s popular Google News service, which clusters the news by stories, then ranks the various media outlets depending on their reach, article length and veracity, and Google Alerts, which proactively informs subscribers of new publications.
  • The Alphabet chief, who has been referred to by Hillary Clinton as a “longtime friend,” added that the experience of “the last year” showed that audiences could not be trusted to distinguish fake and real news for themselves.“We started with the default American view that ‘bad’ speech would be replaced with ‘good’ speech, but the problem found in the last year is that this may not be true in certain situations, especially when you have a well-funded opponent who is trying to actively spread this information,” he told the audience.
  • ...1 more annotation...
  • RT America registered under FARA earlier this month, after being threatened by the US Department of Justice with arrests and confiscations of property if it failed to comply. The broadcaster is fighting the order in court.
1 - 16 of 16
Showing 20 items per page