What kind of Machine Learning/Data Mining packages are available that can easily be scaled on a cluster. For instance H2O is one of them because it's running in Java so it can be easily extended to other machines providing parallelism. What other are popular? or even for general Data munging purposes.

Language wise preferably Python or R. (if other please don't be discouraged to include)

For python (which is my language of preference) I can suggest one of two alternatives.

First, PySpark. As part of the Spark collection of utilities it is very mature and designed to work with hundreds and thousands of nodes. It is extremely stable and has decent API, especially when empowered with Spark-MLlib. Spark has loads of users and resources and comes highly recommended, Nearly the entire industry uses that.

If thousands of nodes is not your biggest concern, and you're familiar with Pandas, you might want to consider Dask for a native python solution, together with sklearn. This is a bit "risky" from a corporate's standpoint but I recommend giving it a try.

• Thank you very much, I did not know about Dask seems a nice framewrok! Aug 23 '16 at 11:45
• An upvote and an "accepted answer" are always a nice way to say "Thank you" ;). No pressure though Aug 23 '16 at 14:53
• I firmly agree with you, I was the one who upvoted you, the reason that I not accepted your answer is because I leave some room for more recommendations especially for the distributed ML part. Again thank you for your proposal! Aug 23 '16 at 15:40

SPARKR, http://spark.apache.org/docs/latest/sparkr.html, is designed to support parallel processing.

Additional resources are summarized here: https://cran.r-project.org/web/views/HighPerformanceComputing.html

• Thank you for the answer, except from these two are there any other ? It seems strange to me that only these two exist. Aug 1 '16 at 19:54