# Data science and MapReduce programming model of Hadoop

What are the different classes of data science problems that can be solved using mapreduce programming model?

Let's first split it into parts.

Data Science is about making knowledge from raw data. It uses machine learning, statistics and other fields to simplify (or even automate) decision making. Data science techniques may work with any data size, but more data means better predictions and thus more precise decisions.

Hadoop is a common name for a set of tools intended to work with large amounts of data. Two most important components in Hadoop are HDFS and MapReduce.

HDFS, or Hadoop Distributed File System, is a special distributed storage capable of holding really large data amounts. Large files on HDFS are split into blocks, and for each block HDFS API exposes its location.

MapReduce is framework for running computations on nodes with data. MapReduce heavily uses data locality exposed by HDFS: when possible, data is not transferred between nodes, but instead code is copied to the nodes with data.

So basically any problem (including data science tasks) that doesn't break data locality principle may be efficiently implemented using MapReduce (and a number of other problems may be solved not that efficiently, but still simply enough).

Let's take some examples. Very often analyst only needs some simple statistics over his tabular data. In this case Hive, which is basically SQL engine over MapReduce, works pretty well (there are also Impala, Shark and others, but they don't use Hadoop's MapReduce, so more on them later).

In other cases analyst (or developer) may want to work with previously unstructured data. Pure MapReduce is pretty good for transforming and standardizing data.

Some people are used to exploratory statistics and visualization using tools like R. It's possible to apply this approach to big data amounts using RHadoop package.

And when it comes to MapReduce-based machine learning Apache Mahout is the first to mention.

There's, however, one type of algorithms that work pretty slowly on Hadoop even in presence of data locality, namely, iterative algorithms. Iterative algorithms tend to have multiple Map and Reduce stages. Hadoop's MR framework reads and writes data to disk on each stage (and sometimes in between), which makes iterative (as well as any multi-stage) tasks terribly slow.

Fortunately, there are alternative frameworks that can both - use data locality and keep data in memory between stages. Probably, the most notable of them is Apache Spark. Spark is complete replacement for Hadoop's MapReduce that uses its own runtime and exposes pretty rich API for manipulating your distributed dataset. Spark has several sub-projects, closely related to data science:

• Shark and Spark SQL provide alternative SQL-like interfaces to data stored on HDFS
• Spark Streaming makes it easy to work with continuous data streams (e.g. Twitter feed)
• MLlib implements a number of machine learning algorithms with a pretty simple and flexible API
• GraphX enables large-scale graph processing

So there's pretty large set of data science problems that you can solve with Hadoop and related projects.

• I'm not so sure about what you say on MapReduce, especially So basically any problem that doesn't break data locality principle may be efficiently implemented using MapReduce. From my understanding, you can only solve problems which can be expressed with the MapReduce pattern. Jul 31 '14 at 9:33
• @fxm: MapReduce framework may be used for pretty much different tasks. For example, Oozie - workflow scheduler for different Hadoop components - has so-called Java action, that simply creates one mapper and runs custom Java code in it. With this approach you can essentially run any code. This, however, won't give you any MR advantages compared to simple Java app. The only way to gain these advantages is to run computations in parallel (using Map) on nodes with data, that is, locally. To summarize: you can run any code with MR, but to obtain performance, you need to hold data locality. Jul 31 '14 at 11:23
• I agree, but what I meant is a developer should not worry about managing data locality (which is managed by hadoop), but rather about expressing the wanted algorithms with the MapReduce pattern. Jul 31 '14 at 12:00
• @fxm: developer ensures locality by using mapper. Roughly speaking, "map(f, data)" means "transfer f() to data nodes and run it locally". If developer doesn't consider locality (e.g. puts all computations to reducer), he loses all advantages of MR. So expressing algo as (efficient) MR job implies utilization of data locality anyway. That said, I would still suggest using more flexible tools than pure MR such as Spark. Jul 31 '14 at 13:29

Data Science has many different sub-areas as described in my post). Nearly for each area, scientists and developer has significant contributions. To learn more about what can be done, please look at following websites:

Also, there are some work on MapReduce + Excel + Cloud combination but I have not found the link.

What are the different classes of Data Science problems ...

Each "classes" is not purely homogeneous problem domain, i.e. some problem cannot be solved via map and reduce approach due to its communication cost, or algorithm behavior. What I mean by behavior is that some problem wants to have control on all data sets instead of chunks. Thus, I refuse to list type of problem "classes".

Do not forget that knowing what MapReduce can do is not enough for Data Science. You should also aware of What MapReduce can't do, too.

There is a paper you should look into:

MapReduce: Distributed Computing for Machine Learning

They distinguish 3 classes of machine-learning problems that are reasonable to address with MapReduce:

1. Single pass algorithms
2. Iterative algorithms
3. Query based algorithms

They also give examples for each class.

map/reduce is most appropriate for parallelizable offline computations. To be more precise, it works best when the result can be found from the result of some function of a partition of the input. Averaging is a trivial example; you can do this with map/reduce by summing each partition, returning the sum and the number of elements in the partition, then computing the overall mean using these intermediate results. It is less appropriate when the intermediate steps depend on the state of the other partitions.