Can R + Hadoop overcome R's memory constraints in any case?

Question

I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.

Can R + Hadoop overcome R's memory constraints in any case?

The answer might be clear as the accepted answer to this question implies but for me it isn't. To be more precise:

Can I use R + Hadoop to fit a model to a really big data set at once. I mean that the entire data is needed for the computation without any independent sub processes which could be parallelized in some way?

I do not see how this can work when using a computer cluster for the computation. In case it is possible: How does it work?

Answer 1

R+Hadoop themselves do not actually give you any massive direct benefit. You could use Hadoop streaming to run parallel R jobs across all the nodes on your Hadoop cluster, but that is reliant on your problem being inherently parallelisable. You need to take advantage of a distributed programming language which will let you run a single program over all the nodes on the Hadoop cluster. There is where something like Apache Spark comes into play. It is important to be pedantic here, as Apache Spark is not INHERENTLY reliant on Hadoop (you can build and compile it standalone without any Hadoop features). Thus why some people argue it is a replacement FOR Hadoop.

Through the SparkR interface, you can access it directly from R just creating the necessary tools (i.e. importing library(sparkR) if I recall correctly, then setting up your sparkContext and SQLContext). This then means, if properly set up, all you have to do to scale your code from 1 machine to 100 machines is add more nodes pointed to the SparkMaster.

Answer 2

To answer your question. I would give an analogous answer to clarify it better. Apache Spark is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run tasks quickly but also utilizes clusters computing for better distribution of the workload.

Similar to that R+ Hadoop also works in the same fashion. What R effectively does is define the tasks or Machine Learning Algorithms to use and it translates over the nodes of the Hadoop cluster to better utilize the parallelization aspect.

Since the usual working of the R language is very RAM dependent this makes the processing time of massive data exponentially high. Usually, R solves this problem by allowing multithreading to take place within the cores so it can run the task as optimized as possible. But as you can tell this provides an upper cap on how much data it can handle. What R + Hadoop does is it provides more nodes so essentially translates to more cores it can use to run the following task.

Edit: As requested more elaboration on the last point

Simply put when looking at just R as a language which is optimally built for data analytics, it utilizes multithreading to perform the required analytical tasks more effectively. This multithreading feature lets you run different tasks as different threads simultaneously so thereby trying to achieve as much as parallelization as possible within the limited capabilities of R.

And to answer your question whether R can perform the same tasks as R+Hadoop provided if there are no time constraints

The answer is theoretically Yes practically No!. Intuitively we might think R can handle such tasks by consuming more time. But that isn't always the case. It depends more on the task.

For example, if R were doing a simple frequency Count of the number occurrences in an array of a few million records you can obviously say that even with a few threads it might be slower than R+Hadoop and finish the task after a certain amount of time. But if the task were to say something like Eigen Value Decomposition of a Matrix, such a transformation requires a lot of calculation at the element level of a matrix. Assuming that the size of that matrix is a few million x few million, this type of calculation will surely cause a memory error using just any normal application as it will bottleneck all the data it has to hold temporarily into the RAM. This bottlenecking problem might cause the whole system to fail at times but the most simple case is that of a memory error.

Using R+Hadoop solves such problems as it lets you distribute the same strategy over multiple computers with their corresponding cores. Hence it lets you handle loads much more easily than running it on the same application, this manner you can also prevent the errors which I mentioned above. This is the power of distributed computing.