Group items tagged

Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.

In my previous article, I introduced the Map/Reduce model as a powerful model for parallelism. However, although Map/Reduce is simple, powerful, and provides a good opportunity to parallelize algorithm, it is based on a rigid procedural structure that requires injection of custom user code and therefore it is not easy to understand the big picture from a high level. You need to drill into the implementation code of the map and reduce functions in order to figure out what is going on.