Then we put the data up, but the problem with Solr was it didn’t have a user interface, so we used Project Blacklight, which is open source software normally used by librarians. We used it for the journalists. It’s simple because it allows you to do faceted search—so, for example, you can facet by the folder structure of the leak, by years, by type of file. There were more complex things—it supports queries in regular expressions, so the more advanced users were able to search for documents with a certain pattern of numbers that, for example, passports use. You could also preview and download the documents. ICIJ open-sourced the code of our document processing chain, created by our web developer Matthew Caruana Galizia.
We also developed a batch-searching feature. So say you were looking for politicians in your country—you just run it through the system, and you upload your list to Blacklight and you would get a CSV back saying yes, there are matches for these names—not only exact matches, but also matches based on proximity. So you would say “I want Mar Cabra proximity 2” and that would give you “Mar Cabra,” “Mar whatever Cabra,” “Cabra, Mar,”—so that was good, because very quickly journalists were able to see… I have this list of politicians and they are in the data!
Gary Edwards on 20 Jun 12Good discussion, but it really deserves a more in-depth thrashing. The basic concept is that a perfect storm of mobility, cloud-computing and HTML5-JavaScript has set the stage for a major, massive shift in application development. The shift from C++ to Java is now being replaced by a greater shift from Java and C++ to JavaScript-JSON-HTML5. Interesting, but i continue to insist that the greater "Perfect Storm" triggered in 2008, is causing a platform shift from client/server computing to full on, must have "cloud-computing". There are three major "waves"; platform shifts in the history of computing at work here. The first wave was "Mainframe computing", otherwise known as server/terminal. The second wave was that of "client/server" computing, where the Windows desktop eventually came to totally dominate and control the "client" side of the client/server equation. The third wave began with the Internet, and the dominance of the WWW protocols, interfaces, methods and formats. The Web provides the foundation for the third great Wave of Cloud-Computing. The Perfect Storm of 2008 lit the fuse of the third Wave of computing. Key to the 2008 Perfect Storm is the world wide financial collapse that put enormous pressure on businesses to cut cost and improve productivity; to do more with less, or die. The survival maxim quickly became do more with less people - which is the most effective form of "productivity". The nature of the collapse itself, and the kind of centralized, all powerful bailout-fascists governments that rose during the financial collapse, guaranteed that labor costs would rise dramatically while also being "uncertain". Think government controlled healthcare. The other aspects of the 2008 Perfect Storm are mobility, HTML5, cloud-computing platform availability, and, the ISO standardization of "tagged" PDF. The mobility bomb kicked off in late 2007, with the introduction of the Apple iPhone. No further explanation needed :) Th
