Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Hadoop divides the input to a MapReduce job into fixed-size pieces called input splits, or just splits. Hadoop creates one map task for each split, which runs the user-defined map function for each record in the split. Having many splits means the time taken to process each split is small compared to the time to process the whole input. So if we are processing the splits in parallel, the processing is better load balanced when the splits are small
Why ?
All bigdata eco system works on something called parallel processing.
We have to process 100gigs of file. If we didnt split the file, then all the 100 gigs should be processed by single JVM(single map).
If we split the file into 1000 parts each of 100mb, then we can process each part with different JVM and apply the map function in less time.
MPP: Massively parallel Processing
