I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert:


What is the "mapreduce_shuffle" option?


1 Answer 1


The mapreduce_shuffle in this config file is part of Plugable Shuffle and Sort. Shuffle and Sort are what connect the mappers to the reducers. A nice graphical representation of this is the following (shuffle is called "copy" in this figure):


The Hadoop arcitecture alllows these steps to be customized:

The pluggable shuffle and pluggable sort capabilities allow replacing the built in shuffle and sort logic with alternate implementations. Example use cases for this are: using a different application protocol other than HTTP such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or replacing the sort logic with custom algorithms that enable Hash aggregation and Limit-N query.

The mapreduce_shuffle parameter is mentioned in the Hadoop Docs, and is the default value for this configuration.


Not the answer you're looking for? Browse other questions tagged or ask your own question.