# Can R + Hadoop overcome R's memory constraints in any case?

I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.

Can R + Hadoop overcome R's memory constraints in any case?

The answer might be clear as the accepted answer to this question implies but for me it isn't. To be more precise:

Can I use R + Hadoop to fit a model to a really big data set at once. I mean that the entire data is needed for the computation without any independent sub processes which could be parallelized in some way?

I do not see how this can work when using a computer cluster for the computation. In case it is possible: How does it work?

R+Hadoop themselves do not actually give you any massive direct benefit. You could use Hadoop streaming to run parallel R jobs across all the nodes on your Hadoop cluster, but that is reliant on your problem being inherently parallelisable. You need to take advantage of a distributed programming language which will let you run a single program over all the nodes on the Hadoop cluster. There is where something like Apache Spark comes into play. It is important to be pedantic here, as Apache Spark is not INHERENTLY reliant on Hadoop (you can build and compile it standalone without any Hadoop features). Thus why some people argue it is a replacement FOR Hadoop.

Through the SparkR interface, you can access it directly from R just creating the necessary tools (i.e. importing library(sparkR) if I recall correctly, then setting up your sparkContext and SQLContext). This then means, if properly set up, all you have to do to scale your code from 1 machine to 100 machines is add more nodes pointed to the SparkMaster.

To answer your question. I would give an analogous answer to clarify it better. Apache Spark is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run tasks quickly but also utilizes clusters computing for better distribution of the workload.

Similar to that R+ Hadoop also works in the same fashion. What R effectively does is define the tasks or Machine Learning Algorithms to use and it translates over the nodes of the Hadoop cluster to better utilize the parallelization aspect.

Since the usual working of the R language is very RAM dependent this makes the processing time of massive data exponentially high. Usually, R solves this problem by allowing multithreading to take place within the cores so it can run the task as optimized as possible. But as you can tell this provides an upper cap on how much data it can handle. What R + Hadoop does is it provides more nodes so essentially translates to more cores it can use to run the following task.

Edit: As requested more elaboration on the last point

Simply put when looking at just R as a language which is optimally built for data analytics, it utilizes multithreading to perform the required analytical tasks more effectively. This multithreading feature lets you run different tasks as different threads simultaneously so thereby trying to achieve as much as parallelization as possible within the limited capabilities of R.