« Will they open source Navision or Great Plains ? | Main | India Inc. is Twice as Fast as Japan Inc »

October 31, 2004

MapReduce

Google team would be presenting in OSDI on the topic pf parallel data processing using large clusters. This abstract throws some light on their much talked about super cluster. -

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a _map_ function that processes a key/value pair to generate a set of intermediate key/value pairs, and a _reduce_ function that merges all intermediate values associated with the same intermediate key.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day

October 31, 2004 in Emerging Technologies | Permalink

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d83455f8ff69e200d8350a168b53ef

Listed below are links to weblogs that reference MapReduce:

Comments

The comments to this entry are closed.