Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert:
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
What is the "mapreduce_shuffle" option?
The mapreduce_shuffle in this config file is part of Plugable Shuffle and Sort. Shuffle and Sort are what connect the mappers to the reducers. A nice graphical representation of this is the following (shuffle is called "copy" in this figure):
The Hadoop arcitecture alllows these steps to be customized:
The pluggable shuffle and pluggable sort capabilities allow replacing the built in shuffle and sort logic with alternate implementations. Example use cases for this are: using a different application protocol other than HTTP such as RDMA for shuffling data from the Map nodes to the Reducer nodes; or replacing the sort logic with custom algorithms that enable Hash aggregation and Limit-N query.
The mapreduce_shuffle parameter is mentioned in the Hadoop Docs, and is the default value for this configuration.
