Group items tagged

This approach can even be useful for applications that are not, strictly speaking, compressed sensing problems, such as the Netflix prize.

Given the enormous popularity of Netflix, even an incremental improvement in the predictive algorithm results in a substantial boost to the company’s bottom line. Recht found that he could accurately predict which movies customers might be interested in purchasing, provided he saw enough products per person. Between 25 and 100 products were sufficient to complete the matrix.

Across every discipline, data sets are getting bigger and more complex, whether one is dealing with medical records, genomic sequencing, neural networks in the brain, astrophysics, historical archives, or social networks. Alessandro Vespignani, a physicist at Northeastern University who specializes in harnessing the power of social networking to model disease outbreaks, stock market behavior, collective social dynamics, and election outcomes, has collected many terabytes of data from social networks such as Twitter, nearly all of it raw and unstructured. “We didn’t define the conditions of the experiments, so we don’t know what we are capturing,” he said.

The impact of data abundance extends well beyond business. Justin Grimmer, for example, is one of the new breed of political scientists. A 28-year-old assistant professor at Stanford, he combined math with political science in his undergraduate and graduate studies, seeing “an opportunity because the discipline is becoming increasingly data-intensive.” His research involves the computer-automated analysis of blog postings, Congressional speeches and press releases, and news articles, looking for insights into how political ideas spread.

But the computer tools for gleaning knowledge and insights from the Internet era’s vast trove of unstructured data are fast gaining ground. At the forefront are the rapidly advancing techniques of artificial intelligence like natural-language processing, pattern recognition and machine learning.

After decades of urban evolution, the world’s major subway systems appear to be converging on an ideal form. On the surface, these core-and-branch systems — evident in New York City, Tokyo, London or most any large metropolitan subway — may seem intuitively optimal. But in the absence of top-down central planning, their movement over decades toward a common mathematical space may hint at universal principles of human self-organization. Understand those principles, and one might “make urbanism a quantitative science, and understand with data and numbers the construction of a city,” said statistical physicist Marc Barthelemy of France’s National Center for Scientific Research.

On the surface, these core-and-branch systems — evident in New York City, Tokyo, London or most any large metropolitan subway — may seem intuitively optimal. But in the absence of top-down central planning, their movement over decades toward a common mathematical space may hint at universal principles of human self-organization.

With equations used to study two-dimensional spatial networks, the class of network to which subways belong, the researchers turned stations and lines to a mathematics of nodes and branches. They repeated their analyses with data from each decade of a subway system’s history, and looked for underlying trends. Patterns emerged: The core-and-branch topology, of course, and patterns more fine-grained. Roughly half the stations in any subway will be found on its outer branches rather than the core. The distance from a city’s center to its farthest terminus station is twice the diameter of the subway system’s core. This happens again and again.