I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.

Can R + Hadoop overcome R's memory constraints in any case?

The answer might be clear as the accepted answer to this question implies but for me it isn't. To be more precise:

Can I use R + Hadoop to fit a model to a really big data set at once. I mean that the entire data is needed for the computation without any independent sub processes which could be parallelized in some way?

I do not see how this can work when using a computer cluster for the computation. In case it is possible: How does it work?


2 Answers 2


R+Hadoop themselves do not actually give you any massive direct benefit. You could use Hadoop streaming to run parallel R jobs across all the nodes on your Hadoop cluster, but that is reliant on your problem being inherently parallelisable. You need to take advantage of a distributed programming language which will let you run a single program over all the nodes on the Hadoop cluster. There is where something like Apache Spark comes into play. It is important to be pedantic here, as Apache Spark is not INHERENTLY reliant on Hadoop (you can build and compile it standalone without any Hadoop features). Thus why some people argue it is a replacement FOR Hadoop.

Through the SparkR interface, you can access it directly from R just creating the necessary tools (i.e. importing library(sparkR) if I recall correctly, then setting up your sparkContext and SQLContext). This then means, if properly set up, all you have to do to scale your code from 1 machine to 100 machines is add more nodes pointed to the SparkMaster.


To answer your question. I would give an analogous answer to clarify it better. Apache Spark is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run tasks quickly but also utilizes clusters computing for better distribution of the workload.

Similar to that R+ Hadoop also works in the same fashion. What R effectively does is define the tasks or Machine Learning Algorithms to use and it translates over the nodes of the Hadoop cluster to better utilize the parallelization aspect.

Since the usual working of the R language is very RAM dependent this makes the processing time of massive data exponentially high. Usually, R solves this problem by allowing multithreading to take place within the cores so it can run the task as optimized as possible. But as you can tell this provides an upper cap on how much data it can handle. What R + Hadoop does is it provides more nodes so essentially translates to more cores it can use to run the following task.

Edit: As requested more elaboration on the last point

Simply put when looking at just R as a language which is optimally built for data analytics, it utilizes multithreading to perform the required analytical tasks more effectively. This multithreading feature lets you run different tasks as different threads simultaneously so thereby trying to achieve as much as parallelization as possible within the limited capabilities of R.

And to answer your question whether R can perform the same tasks as R+Hadoop provided if there are no time constraints

The answer is theoretically Yes practically No!. Intuitively we might think R can handle such tasks by consuming more time. But that isn't always the case. It depends more on the task.

For example, if R were doing a simple frequency Count of the number occurrences in an array of a few million records you can obviously say that even with a few threads it might be slower than R+Hadoop and finish the task after a certain amount of time. But if the task were to say something like Eigen Value Decomposition of a Matrix, such a transformation requires a lot of calculation at the element level of a matrix. Assuming that the size of that matrix is a few million x few million, this type of calculation will surely cause a memory error using just any normal application as it will bottleneck all the data it has to hold temporarily into the RAM. This bottlenecking problem might cause the whole system to fail at times but the most simple case is that of a memory error.

Using R+Hadoop solves such problems as it lets you distribute the same strategy over multiple computers with their corresponding cores. Hence it lets you handle loads much more easily than running it on the same application, this manner you can also prevent the errors which I mentioned above. This is the power of distributed computing.

  • $\begingroup$ Thanks for the answer. But I was not wondering about whether computation time can be reduced (which is obviously clear) but if I could do something with a large data set using Hadoop what I could not do with a single machine (assuming time is not important). If the last part of your post answers this, please make it more precise. $\endgroup$
    – Alex
    Mar 26, 2017 at 11:57
  • $\begingroup$ Please check my edits. I have added more information to better explain your question. $\endgroup$
    – user-116
    Mar 30, 2017 at 18:48
  • $\begingroup$ Was the answer satisfactory?. If then you can confirm the answer or do you want me to explain any other parts in detail ? $\endgroup$
    – user-116
    Apr 2, 2017 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.