Group items tagged

Data journalism at the Guardian: what is it and how do we do it? Simon Rogers: Our 10 point guide to data journalism and how it's changing Share reddit this Comments (2) Data abstract Data journalism. What is it and how is it changing? Photograph: Alamy Here's an interesting thing: data journalism is becoming part of the establishment. Not in an Oxbridge elite kind of way (although here's some data on that) but in the way it is becoming the industry standard. Two years ago, when we launched the Datablog, all this was new. People still asked if getting stories from data was really journalism and not everyone had seen Adrian Holovaty's riposte. But once you've had MPs expenses and Wikileaks, the startling thing is that no-one asks those questions anymore. Instead, they want to know, "how do we do it?"

Open Data Cook BookMaking Open Data Accessible for EveryoneAbout the Cook BookThe open data cook book is collecting recipes for ways to find and use open data, particularly open data of social value - such as open government data, or open data for campaigners and charities. Working with data can seem scary. But it doesn't have to be. There are many different ways to make data useful - and lots of different gadgets to help you. Take a look at the growing list of cook book recipes to find simple step by step ideas for making use of open data. RecipesYou can find a list of the recipes so far here. Drafts, ideas and notesIn the cooks notebook you can find draft notes on using different datasets and sketches that might develop into recipes in future. Get InvolvedFind out how to get involved here or jump right in and create a recipe. Tweet with the #opendatacookbook tag, or bookmark content on del.icio.us 'opendatacookbook ' to share with the project. Join the mailing list to discuss developments. UpdateAfter a brief experiment with Drupal as a CMS for the cook book - we've switched to DokuWiki for a bit to make compiling a list of recipes a lot easier before we work out the best way to run the Cook Book.

8 must-reads detail how to verify information in real-time, from social media, users Craig Silverman by Craig Silverman Published Apr. 27, 2012 7:46 am Updated Apr. 27, 2012 9:23 am Over the past couple of years, I've been trying to collect every good piece of writing and advice about verifying social media content and other types of information that flow across networks. This form of verification involves some new tools and techniques, and requires a basic understanding of the way networks operate and how people use them. It also requires many of the so-called old school values and techniques that have been around for a while: being skeptical, asking questions, tracking down high quality sources, exercising restraint, collaborating and communicating with team members. For example, lots of people talk about how Andy Carvin does crowdsourced verification and turns his Twitter feed into a real time newswire. Lost in the discussion is the fact that Carvin also develops sources and contacts on the ground and stays in touch with them on Skype and through other means. What you see on Twitter is only one part of the process. Some things never go out of style. At the same time, there are new tools, techniques and approaches every journalist should have in their arsenal. Fortunately, several leading practitioners of what I sometimes call the New Verification are gracious and generous about sharing what they know. One such generous lot are the folks at Storyful, a social media curation and verification operation that works with clients such as Reuters, ABC News, and The New York Times, among others. I wrote about them last year and examined how in some ways they act as an outsourced verification service for newsrooms. That was partly inspired by this post from Storyful founder Mark Little: I find it helps to think of curation as three central questions: * Discovery: How do we find valuable social media content? * Verification: How do we make sure we c

Visualization contests around the corner May 25, 2011 to Contests | Comments (3) The best way to learn how to visualize data is to grab a dataset and see what you can do with it. You can read as many tips and tricks as you want, but you're not going to get any better until you actually try. Contests are a fun way to do this. Participate So here are a handful of visualization contests to get your hands dirty. Hey you might even win a couple of thousand dollars. Not that money matters to you, because as well all know, learning is your reward. Hacking Education - A contest for developers and data crunchers. DonorsChoose.org has inspired $80 million in giving from 400,000 donors, helping 165,000 teachers at 43,000 schools, and the donation site has opened up this data. Can do you do something with it? Deadline: June 30, 2011. Data In Sight - A hands-on competition in San Francisco's SoMa district with surprise data sources. Some talks, lunch, dinner, and a 24-hour hackathon. Event date: June 24, 2011 (better to register your team early). Tableau Interactive Viz Contest - This one is coming up the quickest, but is the most straightforward. Plus, you get a t-shirt just for entering. Grab some business, finance, or real estate data and go to town with Tableau Public. Deadline: June 3, 2011.

10 ways to screw up your spreadsheet design Recommend +21 Votes 36 Comments 46Share more + By Susan Harkins June 23, 2011, 8:25 AM PDT Takeaway: How you set up a spreadsheet determines its efficiency, usability, and reliability. Avoiding these pitfalls during the design phase will save you a million headaches. Wrong references, missing values, and invalid data aren't the only things that will ruin a spreadsheet. The development process starts before you do a thing, while you're planning the design. These types of mistakes are worse than bugs because you can't troubleshoot them. All you can do is start over. Here are 10 mistakes to avoid early in the process, when you're still in the decision-making phase.

A good list and read down into the comments; additional good tips there.

Using Overview to analyze 4500 pages of documents on security contractors in Iraq by Jonathan Stray on 02/21/2012 0 This post describes how we used a prototype of the Overview software to explore 4,500 pages of incident reports concerning the actions of private security contractors working for the U.S. State Department during the Iraq war. This was the core of the reporting work for our previous post, where we reported the results of that analysis. The promise of a document set like this is that it will give us some idea of the broader picture, beyond the handful of really egregious incidents that have made headlines. To do this, in some way we have to take into account most or all of the documents, not just the small number that might match a particular keyword search. But at one page per minute, eight hours per day, it would take about 10 days for one person to read all of these documents - to say nothing of taking notes or doing any sort of followup. This is exactly the sort of problem that Overview would like to solve. The reporting was a multi-stage process: Splitting the massive PDFs into individual documents and extracting the text Exploration and subject tagging with the Overview prototype Random sampling to estimate the frequency of certain types of events Followup and comparison with other sources

What is BuzzData? Data should be free-flowing, well-organized and easy to share. Wouldn't it be nice if there was a place where you could store, share and show off your data with just a couple of mouse clicks? BuzzData lets you publish your data in a smarter, easier way. Instead of juggling versions and overwriting files, use BuzzData and enjoy a social network designed for data."

How to use APIs from Twitter, Google & Facebook to find data, ideas Katharine Jarmul by Katharine Jarmul Published Aug. 8, 2011 1:27 pm Updated Aug. 9, 2011 12:02 am As more and more journalists are finding, APIs are a great way to get data for your Web applications and projects. An API, or application programming interface, enables software programs to communicate with one another. (Chrys Wu wrote a helpful intro here.) To give you a better understanding of how they can help you, I've outlined some of the best APIs for finding content and explained how you can use open-source programming tools to glean information from them.

Charts and graphs are the best ways to represent information and relationship between two interlinked entities. Not only do charts and graphs inform visitors about the trend or relationship you want to show them but also add a visual connection with the visitors. Several online tools are available that help you create comprehensively designed flowcharts and graphs that worth a thousands words. Check them out and let us know what you feel about these tools. http://www.smashingapps.com/2011/08/26/45-free-online-tools-to-create-charts-diagrams-and-flowcharts.html

Welcome to Graphviz Available translations:Belorussian What is Graphviz? Graphviz is open source graph visualization software. Graph visualization is a way of representing structural information as diagrams of abstract graphs and networks. It has important applications in networking, bioinformatics, software engineering, database and web design, machine learning, and in visual interfaces for other technical domains. Features The Graphviz layout programs take descriptions of graphs in a simple text language, and make diagrams in useful formats, such as images and SVG for web pages, PDF or Postscript for inclusion in other documents; or display in an interactive graph browser. (Graphviz also supports GXL, an XML dialect.) Graphviz has many useful features for concrete diagrams, such as options for colors, fonts, tabular node layouts, line styles, hyperlinks, rolland custom shapes.

Matthew Ericson « Illustrator MultiExporter script: Now with JPG and EPS When Maps Shouldn't Be Maps View full interactive map on nytimes.com » Often, when you get data that is organized by geography - say, for example, food stamp rates in every county, high school graduation rates in every state, election results in every House district, racial and ethnic distributions in each census tract - the impulse is since the data CAN be mapped, the best way to present the data MUST be a map. You plug the data into ArcView, join it up with a shapefile, export to Illustrator, clean up the styles and voilà! Instant graphic ready to be published. And in many cases, that's the right call.

Mapping Sheets Another favorite Google Drive add-on of mine is definitely Mapping Sheets. With this add-on, you can make better use of any geographical data you may have in your spreadsheet. If you've ever wanted an easy way to quickly plot locations from your data onto a Google Map, this is it. drive addons8   5 Google Drive Add ons You Need To Use Using it is ridiculously easy. Just make sure you've got a list of addresses and other related data in your sheet, and then trigger this add-on. You'll see a form where you tell it what columns in your sheet to use for creating the map. drive addons9   5 Google Drive Add ons You Need To Use Once you submit it, the add-on creates the map right in the sheet for you so that you have a useful visualization of all of that data.

"ReConstitution 2012, a fun experiment by Sosolimited, processes transcripts from the presidential debates, and recreates them with animated words and charts. Part data visualization, part experimental typography, ReConstitution 2012 is a live web app linked to the US Presidential Debates. During and after the three debates, language used by the candidates generates a live graphical map of the events. Algorithms track the psychological states of Romney and Obama and compare them to past candidates. The app allows the user to get beyond the punditry and discover the hidden meaning in the words chosen by the candidates. As you let the transcript run, numbers followed by their units (like "18 months") flash on the screen, and trigger words for emotions like positivity, negativity, and rage are highlighted yellow, blue, and red, respectively. You can also see the classifications in graph form. There are a handful of less straightforward text classifications for truthy and suicidal, which are based on linguistic studies, which in turn are based on word frequencies. These estimates are more fuzzy. So, as the creators suggest, it's best not to interpret the project as an analytical tool, and more of a fun way to look back at the debate, which it is. It's pretty fun to watch. Here's a short video from Sosolimited for more on how the application works: "

What is it? A social analytics platform which contains over 1,000 days of tweets (all 70 billion of them), Facebook activity and blog posts. How is it of use to journalists? "Journalists can easily develop real-time insights into any story from Playground," PeopleBrowsr UK CEO Andrew Grill explains. Complex keyword searches can be divided by user influence, geolocation, sentiment, and virtual communities of people with shared interests and affinities. These features - and many more - let reporters and researchers easily drill down to find the people and content driving the conversation on social networks on any subject. Playground lets you use the data the way you want to use it. You can either export the graphs and tables that the site produces automatically or export the results in a CSV file to create your own visualisations, which could potentially make it the next favourite tool of data journalists. Grill added: The recent launch of our fully transparent Kred influencer platform will make it faster and easier for journalists to find key influencers in a particular community. You can give Playground a try for the first 14 days before signing up for one of their subscriptions ($19 a month for students and journalists, $149 for organisations and companies).

Mining of Massive Datasets The book has now been published by Cambridge University Press. A hardcopy can be obtained Here. By agreement with the publisher, you can still download it free from this page. Cambridge Press does, however, retain copyright on the work, and we expect that you will acknowledge our authorship if you republish parts or all of it. We are sorry to have to mention this point, but we have evidence that other items we have published on the Web have been appropriated and republished under other names. It is easy to detect such misuse, by the way, as you will learn in Chapter 3. --- Anand Rajaraman (@anand_raj) and Jeff Ullman Downloads Download the Complete Book (340 pages, approximately 2MB) Download chapters of the book: Preface and Table of Contents Chapter 1 Data Mining Chapter 2 Large-Scale File Systems and Map-Reduce Chapter 3 Finding Similar Items Chapter 4 Mining Data Streams Chapter 5 Link Analysis Chapter 6 Frequent Itemsets Chapter 7 Clustering Chapter 8 Advertising on the Web Chapter 9 Recommendation Systems Index

Statistical Reasoning 1 http://ocw.jhsph.edu/index.cfm/go/viewCourse/course/StatisticalReasoning1/coursePage/index/ Most people could probably use a bit of a refresher on statistical reasoning and its methods, and this free course from Johns Hopkins University is a great way to get started on the road back to statistical literacy. The course was originally taught by John McGready and provides "a broad overview of biostatistical methods and concepts used in the public health sciences." Users will find that the home page includes links to the course syllabus, schedule, lecture materials, readings, and additional assignments. The Lecture Materials area includes course notes from the seven modules here. The topics include "Describing Data," "An Introduction to Hypothesis Testing," and "When Time Is of Interest: The Case for Survival Analysis." Visitors can also take advantage of the assignments, which correspond to the readings and the lecture materials. The site is completed by the Other Resources area, which includes a special lecture on the software package Stata and a flowchart designed to help students learn how to choose the correct statistical procedure for the task at hand. [KMG]

Welcome to the Improving data visualisation for the public sector project Good data visualisation can help users explore and understand the patterns and trends in data, and also communicate that understanding to others to help them make robust decisions based on the data being presented. This site supports public sector researchers improve the way that they visualise data, by providing good practice examples and case studies, practical and step-by-step guides on how to visualise data, and links to more detailed resources. http://www.improving-visualisation.org

The web site batchgeo.com provides an easy-to-use web interface for creating interactive Google maps. If you have names and addresses and other information, you can quickly create on online Google map with up to 2500 points. "Maps tell a story, see what you're missing BatchGeo is simply the fastest way to create google maps from your address lists. It accepts addresses, intersections, cities, states, and postal codes. We do the hard work of figuring out where all your data lives in the real world."

Constructing the Open Data Landscape Posted on September 7, 2011 by Nicola Hughes In an article in today's Telegraph regarding Francis Maude's Public Data Corporation, Michael Cross asks: "What makes the state think it can be at the cutting edge of the knowledge economy". He writes in terms of market and business share, giving the example of the satnav market worth over $100bn a year yet it's based on free data from the US Government's GPS system. He credits the internet revolution for transforming public sector data into 'cashable proposition'. We, along with many other start-ups, foundations and civic coding groups, are part of this 'geeky world' of Open Data. So we'd like to add our piece concerning the Open Data movement. Michael has the right to ask this question because there is this constant custodial battle being fought every day, every scrape and every script on the web for the rights to data. So let me tell you about the geeks' take on Open Data.

They Rule Overview They Rule aims to provide a glimpse of some of the relationships of the US ruling class. It takes as its focus the boards of some of the most powerful U.S. companies, which share many of the same directors. Some individuals sit on 5, 6 or 7 of the top 1000 companies. It allows users to browse through these interlocking directories and run searches on the boards and companies. A user can save a map of connections complete with their annotations and email links to these maps to others. They Rule is a starting point for research about these powerful individuals and corporations. Context A few companies control much of the economy and oligopolies exert control in nearly every sector of the economy. The people who head up these companies swap on and off the boards from one company to another, and in and out of government committees and positions. These people run the most powerful institutions on the planet, and we have almost no say in who they are. This is not a conspiracy, they are proud to rule, yet these connections of power are not always visible to the public eye. Karl Marx once called this ruling class a 'band of hostile brothers.' They stand against each other in the competitve struggle for the continued accumulation of their capital, but they stand together as a family supporting their interests in perpetuating the profit system as whole. Protecting this system can require the cover of a 'legitimate' force - and this is the role that is played by the state. An understanding of this system can not be gleaned from looking at the inter-personal relations of this class alone, but rather how they stand in relation to other classes in society. Hopefully They Rule will raise larger questions about the structure of our society and in whose benefit it is run. The Data We do not claim that this data is 100% accurate at all times. Corporate directors have a habit of dying, quitting boards, joining new ones and most frustratingly passing on their name

I think this data must be very useful to the people in Occupy Wall Street