Google's MapReduce algorithm

The New York Times had a good article a while back about Google's hardware and software infrastructure.  It said that one of their big innovations was an algorithm called MapReduce:

One such program, called MapReduce, is based on ideas discussed in computer science literature for decades, according to Urs Hölzle, Google's senior vice president for operations. "What surprised us was how useful it turned out to be in our environment," he said.

MapReduce, he said, "allows Joe Schmo software engineer to process large amounts of data and take advantage of our infrastructure."

Mr. Arnold, the consultant, said these tools created a significant cost advantage. "If you talk to guys who work in massively parallel computing operations, as much as 30 percent of their coding time is spent trying to figure out how to get the thing to run," he said. Google "has figured out how they can reduce a lot of the hassle and work of creating parallel applications."

Mr. Gates acknowledged that MapReduce was a significant technology, but he asserted that Microsoft was building its own parallel processing software, opening another front in the technological war between the two companies.

"They did MapReduce; we have this thing called Dryad that's better," Mr. Gates said. "But they'll do one that's better."

Anything with such a snarky quote from Bill Gates has got to be interesting.  So here's the paper that describes it.  It is an interesting approach to parallel computing, well worth a read.

Comments

[priyank jain]

sir it was awsome...
bt cn u provide me some more material on Map Reduce methodology....