0
$\begingroup$

What kind of Machine Learning/Data Mining packages are available that can easily be scaled on a cluster. For instance H2O is one of them because it's running in Java so it can be easily extended to other machines providing parallelism. What other are popular? or even for general Data munging purposes.

Language wise preferably Python or R. (if other please don't be discouraged to include)

|improve this question
$\endgroup$
1
$\begingroup$

For python (which is my language of preference) I can suggest one of two alternatives.

First, PySpark. As part of the Spark collection of utilities it is very mature and designed to work with hundreds and thousands of nodes. It is extremely stable and has decent API, especially when empowered with Spark-MLlib. Spark has loads of users and resources and comes highly recommended, Nearly the entire industry uses that.

If thousands of nodes is not your biggest concern, and you're familiar with Pandas, you might want to consider Dask for a native python solution, together with sklearn. This is a bit "risky" from a corporate's standpoint but I recommend giving it a try.

|improve this answer
$\endgroup$
  • $\begingroup$ Thank you very much, I did not know about Dask seems a nice framewrok! $\endgroup$ – Grzegorz Aug 23 '16 at 11:45
  • $\begingroup$ An upvote and an "accepted answer" are always a nice way to say "Thank you" ;). No pressure though $\endgroup$ – NirIzr Aug 23 '16 at 14:53
  • 1
    $\begingroup$ I firmly agree with you, I was the one who upvoted you, the reason that I not accepted your answer is because I leave some room for more recommendations especially for the distributed ML part. Again thank you for your proposal! $\endgroup$ – Grzegorz Aug 23 '16 at 15:40
1
$\begingroup$

SPARKR, http://spark.apache.org/docs/latest/sparkr.html, is designed to support parallel processing.

Additional resources are summarized here: https://cran.r-project.org/web/views/HighPerformanceComputing.html

|improve this answer
$\endgroup$
  • $\begingroup$ Thank you for the answer, except from these two are there any other ? It seems strange to me that only these two exist. $\endgroup$ – Grzegorz Aug 1 '16 at 19:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.