Skip to main content

Home/ Math-a-manics/ Group items matching "movies" in title, tags, annotations or url

Group items matching
in title, tags, annotations or url

Sort By: Relevance | Date Filter: All | Bookmarks | Topics Simple Middle
Peter Kronfeld

Scientific Data Has Become So Complex, We Have to Invent New Math to Deal With It - Wired Science - 0 views

www.wired.com/...all

math data netflix

shared by Peter Kronfeld on 10 Oct 13 - No Cached
  • This approach can even be useful for applications that are not, strictly speaking, compressed sensing problems, such as the Netflix prize.
    • Peter Kronfeld
      Peter Kronfeld on 10 Oct 13
       
      Took 2006 - 2009 to accomplish, by an "international team of statisticians, machine learning experts and computer engineers"
  • Given the enormous popularity of Netflix, even an incremental improvement in the predictive algorithm results in a substantial boost to the company’s bottom line. Recht found that he could accurately predict which movies customers might be interested in purchasing, provided he saw enough products per person. Between 25 and 100 products were sufficient to complete the matrix.
  • Across every discipline, data sets are getting bigger and more complex, whether one is dealing with medical records, genomic sequencing, neural networks in the brain, astrophysics, historical archives, or social networks. Alessandro Vespignani, a physicist at Northeastern University who specializes in harnessing the power of social networking to model disease outbreaks, stock market behavior, collective social dynamics, and election outcomes, has collected many terabytes of data from social networks such as Twitter, nearly all of it raw and unstructured. “We didn’t define the conditions of the experiments, so we don’t know what we are capturing,” he said.
  • ...4 more annotations...
  • It wasn’t the size of the data set that was daunting; by big data standards, the size was quite manageable. It was the sheer complexity and lack of formal structure that posed a problem.
  • calculus lets you take a lot of simple models and integrate them into one big picture.” Similarly, Coifman believes that modern mathematics — notably geometry — can help identify the underlying global structure of big datasets.
  • The key to the technique’s success is a concept known as sparsity, which usually denotes an image’s complexity, or lack thereof. It’s a mathematical version of Occam’s razor: While there may be millions of possible reconstructions for a fuzzy, ill-defined image, the simplest (sparsest) version is probably the best fit. Out of this serendipitous discovery, compressed sensing was born.
  • Using compressed sensing algorithms, it is possible to sample only 100,000 of, say, 1 million pixels in an image, and still be able to reconstruct it in full resolution — provided the key elements of sparsity and grouping (or “holistic measurements”) are present. It is useful any time one encounters a large dataset in which a significant fraction of the data is missing or incomplete.
1 - 2 of 2
Showing 20▼ items per page