Group items tagged

Perception: Gestalt Laws Home → Data visualization for Excel users → Perception: Gestalt Laws Every chart starts with a table. We transcribe this table into a visual representation of distances between data points: the "origin chart". That's when our "eye-brain system" starts making assumptions. It assumes that data points are somewhat related, even if they are not:

The primary World Bank collection of development indicators, compiled from officially-recognized international sources. It presents the most current and accurate global development data available, and includes national, regional and global estimates. Complete dataset available from: http://data.worldbank.org/data-catalog/world-development-indicators

Eurostat was established in 1953 to meet the requirements of the Coal and Steel Community. Over the years its task has broadened and when the European Community was founded in 1958 it became a Directorate-General (DG) of the European Commission. Eurostat's key role is to supply statistics to other DGs and supply the Commission and other European Institutions with data so they can define, implement and analyse Community policies. The result: Eurostat offers a whole range of important and interesting data that governments, businesses, the education sector, journalists and the public can use for their work and daily life. With the development of Community policies, Eurostat's role has changed. Today, collecting data for EMU and developing statistical systems in candidate countries for EU membership are more important than ten years ago.

In this piece, I'll try to demystify JSON so that you can at least recognize it when you come across it. Again, it is just a data format. Reading and understanding JSON doesn't require programming. But after you see how JSON is used, you'll realize why it might be worth your while to learn some programming.

"Abstract Much of the data underlying global poverty and inequality estimates is not in the public domain, but can be accessed in small pieces using the World Bank's PovcalNet online tool. To overcome these limitations and reproduce this database in a format more useful to researchers, we ran approximately 23 million queries of the World Bank's web site, accessing only information that was already in the public domain. This web scraping exercise produced 10,000 points on the cumulative distribution of income or consumption from each of 942 surveys spanning 127 countries over the period 1977 to 2012. This short note describes our methodology, briefly discusses some of the relevant intellectual property issues, and illustrates the kind of calculations that are facilitated by this data set, including growth incidence curves and poverty rates using alternative PPP indices. The full data can be downloaded at www.cgdev.org/povcalnet. "

Google Correlate lets you see how your data relates to search queries Posted: 25 May 2011 11:27 AM PDT Influenza search - Google Correlate A while back, Google showed how Influenza outbreaks correlated to searches for flu-related terms with Google Flu Trends. It helped researchers and policy-makers estimate flu activity much sooner than with previous methods. Google Correlate is the evolution of Flu Trends in that now you can correlate search trends with not just flu cases, but with your own data or other search queries. The above, which you already know about, matches flu cases with searches for "treatment for flu." Similarly, the search phrase that correlates highest with "Toyota for sale" is "used Hyundai," as shown below. You can also see how your data is related geographically. For example, annual rainfall (left) strongly correlates with searches for "disney vacation package." Although, it looks like distance is a strong factor in the latter, which should be a reminder that correlation is different from causation. Google is careful to point this out in their FAQ and explanation of the tool. Nevertheless, it's fun to poke around and sometimes see the non-sensical correlations. For example, the strongest correlation with "flowingdata" is "how to scan a document," because the growth rates of both seem similar. There's also a search by drawing function. You draw a time series, and Correlate finds terms that best match that trend. In the below chart, I drew a line (blue) that had steady growth, but plateaued towards present day. What weird correlations can you find? [Google Correlate]

How to make searchable, Web-based Google charts Michelle Minkoff by Michelle Minkoff Published June 3, 2011 12:01 am Updated June 2, 2011 10:22 pm A lot of data visualization requires the technical expertise of a programmer and skills that take time and resources to develop. A rise in free tools, however, has made it easier to make interactive graphs in charts, whether you're a designer, developer, Web producer or hobbyist. The Google Visualization API, for instance, gives you options without making the work too complicated. I've created a tutorial below to help you make simple, Web-based Google charts. (You can click on any of the screenshots to go to a larger version.) In the first example, we'll craft an interactive bar chart that compares the numbers of tornado-related deaths in the United States throughout the past four years. We'll use data from the National Oceanic and Atmospheric Administration (NOAA), which can be found here. (You can download a cleaned version of this data here, formatted as a comma-delimited file, CSV.) http://www.poynter.org/how-tos/newsgathering-storytelling/126595/how-to-make-simple-web-based-google-charts

Hello, this is the Open Flash Chart project. Note: "Open Flash Chart 2" is LGPL. OK, Open Flash Chart 1.x was great and it works like a dream. But I made some little mistakes which over time grew and anyoyed me and made the source code weird. So I decided it was time to re-jigger the code and make it pretty again. The big change is moving the data format to JSON. This has made a big difference and has allowed some pretty cool new features. While I was hacking away at the source code I moved it all to Actionscript 3, and used Adobe Flex to compile it. This means everything is open source. If you want to make changes to the charts all you need is laid out in these instructions. Just because there is a new version doesn't make V 1.x obsolete. You can use both versions at the same time so leave your current working code in V 1.x and make all the new charts using which ever version you find easier to use. Why is V2 better? Well it uses JSON as the file format and this means you can do cool stuff like Grant Slender has: http://code.google.com/p/ofcgwt/ If you like Open Flash Chart and want to see it continue, please help Donate some money :-) Blog about it (promotion takes up about a third of my time) Write a cool library Really. You can make a massive difference to the project! Need help choosing reseller hosting for your charts? Make sure you read reliable web hosting reviews. Why choose Open Flash Chart? This is a little gentle propaganda for the project. Like all opinions, disregard it and make up your own mind. Edge cases such as tooltips encourage user interactivity and data exploration what happens to the tooltip when two points are in the same position? you can re-size the charts missing data save the chart as an image You can highlight or emphasize one (or many) points PC Pro loves open flash chart. Server Side Helper Libraries PHP, Perl, Python, Ruby, .NET, Google Web Toolkit and JAVA. Libraries. Next: Che

An interesting example of yet another "next generation" data analysis and presentation tool. You can see the demos at http://www.lavastorm.com/ Emphasis is on visualizing the data analytic method while doing the analysis.

We are committed to equal access to technology. Our software is freely available, and anyone may share our technology and modify it to suit their needs - all without asking our permission. Benetech created Martus and Analyzer specifically for human rights data collection, coding and processing. These tools include cryptographic security features and flexible data structures that can be adapted to the needs of each human rights project. By releasing our software as open source, we participate in the technological community where tools can be audited and improved by others, as well as enabling widespread access to our ideas.

Google Refine is a power tool for working with messy data, primarily for * detecting and fixing inconsistencies * transforming data from one structure or format to another * connecting names within your data to name registries (databases) Use Google Refine when you need something ... * more powerful than a spreadsheet * more interactive and visual than scripting * more provisional / exploratory / experimental / playful than a database

kdmc data visualization tutorials KDMC produces a wealth of digital media tutorials to support our training sessions and classes. While the focus of some tutorials is on technology and journalism, most are general enough to be of use to anyone.

A very good collection of dataviz tips and tools

The Investigative Dashboard (ID) is a work in progress, that is designed to showcase the potential for collaboration and data-sharing between investigative reporters across the world. The initiative is spearheaded by the Organized Crime and Corruption Reporting Project, the Romanian Center for Investigative Journalism, the Forum for African Investigative Reporters and the International Center for Journalists, and will expand to include other institutional members of the Global Investigative Journalism Network. The project is coordinated by Paul Cristian Radu (of OCCRP and CRJI) and Justin Arenstein (of FAIR) and was developed while both were in residence at Stanford University as Knight fellows. The John S. Knight Fellowships for Professional Journalists made possible the ID by providing access to the know-how of co-fellow journalists and of experts at Stanford University and in Silicon Valley. This first iteration of the ID website shares detailed methodologies, resources, and links for journalists to track money, shareholders, and company ownership across international borders. It also shares video tutorials, and other tools, to help journalists navigate often rapidly evolving data-sources. Future versions of ID will offer more advanced collaborative workspaces, data-archives, and discounted (or, where possible, free) access to expensive or proprietary research services. But, perhaps most importantly, the ID will campaign for investigative centres across the world to collaborate with each other to improve the depth and impact of their reportage.

Mining of Massive Datasets The book has now been published by Cambridge University Press. A hardcopy can be obtained Here. By agreement with the publisher, you can still download it free from this page. Cambridge Press does, however, retain copyright on the work, and we expect that you will acknowledge our authorship if you republish parts or all of it. We are sorry to have to mention this point, but we have evidence that other items we have published on the Web have been appropriated and republished under other names. It is easy to detect such misuse, by the way, as you will learn in Chapter 3. --- Anand Rajaraman (@anand_raj) and Jeff Ullman Downloads Download the Complete Book (340 pages, approximately 2MB) Download chapters of the book: Preface and Table of Contents Chapter 1 Data Mining Chapter 2 Large-Scale File Systems and Map-Reduce Chapter 3 Finding Similar Items Chapter 4 Mining Data Streams Chapter 5 Link Analysis Chapter 6 Frequent Itemsets Chapter 7 Clustering Chapter 8 Advertising on the Web Chapter 9 Recommendation Systems Index

WHAT WE DO We build software that allows organizations to make sense of massive amounts of disparate data. We solve the technical problems, so they can solve the human ones. Combating terrorism. Prosecuting crimes. Fighting fraud. Eliminating waste. From Silicon Valley to your doorstep, we deploy our data fusion platforms against the hardest problems we can find, wherever we are needed most.

Tom Torok (NYT) writes: After years of looking down my nose at Excel because of its limitations, I have to say that I'm very impressed with Excel 2010 when used with a free Microsoft add-in called PowerPivot. http://office.microsoft.com/en-us/excel/download-powerpivot-HA101959985.aspx In a PowerPivot tutorial (link below), I imported eight tables  from several sources and joined them - yes, you can join relational data. It uses some magical data compression that allows for lightning fast sorts, filters and calculated fields. The largest table in the tutorial has about 2 million rows. A calculated field on that table took seconds. A did a pivot table on the table and the answers appeared as soon as I selected the fields. In one of  the training videos (http://www.powerpivot.com/) an MS guy works with a 101 million-record table on his laptop. It's really amazing. http://powerpivotsdr.codeplex.com/ If you install, be sure to read the prerequisites or you'll be installing and uninstalling both PowerPivot and Excel. I'm running it on a 32-bit XP machine (it won't run on a 64-bit XP but will work on Windows 7 64-bit). The tutorial is for a Windows 7 setup, but there are items in the menu bar that match the reference to the tutorial's ribbon. I noticed that if I call up an xlsx by double clicking on a file in Windows Explorer that PowerPivot is not enabled in the ribbon. If you call up a file from within Excel 2010 everything works as advertised.Regards, TT  

A free search engine for data sets published by governments, private companies and other organizations. It now indexes 255,180 datasets from many sources.

Data-drivenjournalism:Whatistheretolearn?Apaperonthedata-drivenjournalismroundtableheldinAmsterdamon24August2010.Withadditionalmaterialondatatools,DDJinnovators,andrecommendedwebsitesandarticles.Theimmediategoalsaretoimproveaccessforinterestedjournalistsandtoidentifytrainingneedsforthefuture

What is it? A social analytics platform which contains over 1,000 days of tweets (all 70 billion of them), Facebook activity and blog posts. How is it of use to journalists? "Journalists can easily develop real-time insights into any story from Playground," PeopleBrowsr UK CEO Andrew Grill explains. Complex keyword searches can be divided by user influence, geolocation, sentiment, and virtual communities of people with shared interests and affinities. These features - and many more - let reporters and researchers easily drill down to find the people and content driving the conversation on social networks on any subject. Playground lets you use the data the way you want to use it. You can either export the graphs and tables that the site produces automatically or export the results in a CSV file to create your own visualisations, which could potentially make it the next favourite tool of data journalists. Grill added: The recent launch of our fully transparent Kred influencer platform will make it faster and easier for journalists to find key influencers in a particular community. You can give Playground a try for the first 14 days before signing up for one of their subscriptions ($19 a month for students and journalists, $149 for organisations and companies).