Group items tagged

How Big Data Became So Big By STEVE LOHR Published: August 11, 2012 This has been the crossover year for Big Data - as a concept, as a term and, yes, as a marketing tool. Big Data has sprung from the confines of technology circles into the mainstream. First, here are a few, well, data points: Big Data was a featured topic this year at the World Economic Forum in Davos, Switzerland, with a report titled "Big Data, Big Impact." In March, the federal government announced $200 million in research programs for Big Data computing.

Open data initiatives such as Linked Data have begun to see adoption (e.g. NYT), with the objective of opening up and creating relationships between databases. This presents opportunities to break down the barriers that separate individual websites and their content, creating unprecedented opportunities for the reuse and combination of different data sources. A common critique of apps has been the walled-garden approach to storing data and communicating with other apps-open data promises to change that. As a consequence we will begin to see more apps that aggregate a multitude of sources, rather than remaining tethered to any particular set of content (though those will likely continue to exist). These will be apps designed for a particular scenario and user experience-their primary point of differentiation. They will use whatever data sources are available in order to best satisfy the goals of whatever experience or scenario is being designed.

"Seattle big data startup SpaceCurve announced today that it has set a real-time Big Data performance record for ingesting streaming data and spitting out numbers immediately for queries. SpaceCurve ran a full-blown analytics test on Tweet records that averaged 2,500 bytes in size and included location, user and time information. The rate of operation was done at millions of records per second, hundreds of billions of records per day and petabytes of data per day - a new record, according to SpaceCurve."

From basic statistics to machine learning and new ways to think about visualization, the Data Science Starter Kit gives you the tools you need to get started with data. If you haven't yet taken the leap, why wait? And if you're already experienced with data, the Starter Kit will push you further. The package includes (8) titles on R, basic statistics and data analysis, Python, machine learning, and visualization. This kit includes everything you need from analysis, visualization, to management.

"Do you want to get more control over your data? Anyone who works with data can benefit from learning SQL, whether you're an online campaigner, a voterfile manager, an analyst, a pollster, or anyone else who works extensively with data."

The advantage of NodeXL, particularly for graphing Twitter communities, is it has built-in features for grabbing the data for you. Not only that the coding is clever enough to handle the data collection for mere mortals, so when you hit your rate limit NodeXL waits until it should be able to get more data. NodeXL also has "built-in connections for getting networks from Flickr, YouTube, and your local email. Additional importers for Exchange Email, Facebook, and Hyperlink networks are available".

Quantity has a quality all its own, as the saying goes. That's especially true of information. The more you've got, the more useful it becomes. Knowing sensitive facts about one person, or a dozen, may be trivially useful. But analyze the same facts about 100 million people, and you can cure diseases, win elections, or earn billions of dollars, because unpredictable insights emerge when you turn computers loose on vast storehouses of information. There's a nickname for the concept: "big data." It's one of the buzzwords of corporate executives, tech-savvy politicians, and worried civil libertarians. If you want to know what they're all talking about, then "Big Data'' is the book for you, a comprehensive and entertaining introduction to a very large topic. By analyzing huge amounts of information, it's possible to discover patterns and relationships that up to now have been invisible to us. In this way, we can find new solutions to tough problems, and opportunities we'd never otherwise have suspected.

As I have said before, I think the term "data science" is a bit of a misnomer, but I was very hopeful after this discussion; mostly because of the utter lack of agreement on what a curriculum on this subject would look like. The difficulty in defining these skills is that the split between substance and methodology is ambiguous, and as such it is unclear how to distinguish among hackers, statisticians, subject matter experts, their overlaps and where data science fits.

Big data might very well be the tech buzz word of the year. And just how hot is it? Consider this:  Data scientist Stephen Purpura - an expert in artificial intelligence, machine learning and predictive analytics who is studying for his PhD in information sciences at Cornell University - has received no fewer than 45 job offers in recent months. And they just aren't any fly-by-night offers, with some rolling in from big-name companies touting salaries of $300,000 or more.

Ecology maintains the spatial datasets described here in order to better describe the diverse natural and cultural environment that we live and work in. As a public service, we have made some of our data available for viewing and downloading here.

Cogito semantic technology The material analyzed through the intelligence process is mostly unstructured: communications, conversations, data, news, etc., and rarely has an organized form, such as a database structure. As a consequence, the activities of analysis and processing can be highly resource consuming, and prone to error. The work of analysts and knowledge workers remains essential, and technology cannot fully replace it (and never will). Nevertheless, with new generation software based on semantic analysis, it is now possible to understand data automatically as well as to support reasoning and stimulate intuition.

SNA package used for "ordinary" data (one line of data for each case) and for network data (two files - "nodes" and "links" )

What makes a tweet go viral online? And what does a viral trend actually look like? Those are a couple of the questions that can be answered by a new Microsoft Research project, called Viral Search. The company's researchers are showing the project this week at an internal gathering this week in Redmond. The program crunches large amounts of data from Twitter (and potentially Facebook and other platforms in the future) to analyze and display patterns of distribution on the social network.

"A Seattle data technology startup is taking the phrase "know your customer" to a whole new level. Versium Analytics today introduced a new analytics platform called LifeData. Led by former InfoSpace executives Chris Matty and Kevin Marcus, the 12-month-old company is trying to help organizations learn more about their customers by using a patent-pending technology that verifies and cross-indexes large sets of data."

Meteorologists may become better at their jobs thanks to your smartphone. Atmospheric scientists at the University of Washington are collecting data from people using PressureNet, a free app that works on certain Android devices.

Discourse Network Analyzer (DNA) is a Java-based application for qualitative category-based content analysis. It serves two purposes: coding statements of actors into categories, and converting these structured data into networks that are readable by UCINET, visone and other network-analytic software packages. The software can extract either bipartite (affiliation) networks or adjacency networks. It is complementary to semantic network analysis applications because it neither relies on algorithms for automatic text processing, nor does it focus on the internal meaning or mental representation of a single text or document. Instead, tags are applied to the text data by manual inspection, thus rendering it more flexible, yet at the same time more work-intensive. DNA can be used to code a large body of text documents and then convert them into graphs. The application is currently being developed, tested and heavily used in my dissertation research project on German pension politics. Updates will be posted here as soon as something has been published. If you use DNA, I would love to hear from you about your project (if possible, via the DNA-help mailing list). For more information about the software, please consult the documentation or obtain a free copy from the download page.

GeoPlatform.gov/gulfresponse is a new online tool that provides you with near-real time information about the response effort. Developed by NOAA with the EPA, U.S. Coast Guard, and the Department of Interior, the site offers you a "one-stop shop" for spill response information. The site integrates the latest data the federal responders have about the oil spill's trajectory with fishery area closures, wildlife data and place-based Gulf Coast resources - such as pinpointed locations of oiled shoreline and current positions of deployed research ships - into one customizable interactive map.

Time-series network data to a dynamic animation using Visone (www.leydesdorff.net/visone).

See relationships among data points: Network Diagram; Scatterplot; Matrix Chart Compare a set of values: Bar Chart; Block Histogram; Bubble Chart Track rises and falls over time: Line Graph; Stack Graph; Stack Graph for Categories See the parts of a whole: Pie Chart; Treemap; Treemap for Comparisons Analyze a text: Word Tree; Tag Cloud; Word Cloud Generator; Phrase Net See the world: Massachusetts Map; World Map; US County Map; New Jersey Map