2
$\begingroup$

I’m using Spark Scala for multiclass classification, and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, Multinomial and Bernoulli are supported, where I would need to use Gaussian, and LogisticRegressionWithLBFGS would not be suitable either.

I know that in Python, you can integrate sci-kit learn with Spark, but are there any options when using Scala?

Interested to hear people’s thoughts on this.

$\endgroup$
  • $\begingroup$ Welcome to DataScience.SE! I would recommend extending the existing Spark classes. $\endgroup$ – Emre May 27 '16 at 16:05
1
$\begingroup$

The way you can use scikit-learn is basically broadcasting your data to the workers and then do different folds of crossvalidation or different parameter settings in your grid_search on different workers. That is all that the scikit-learn package in pySpark does as far as I know. This is similar to a normal mapping. Implementing this should be relatively easy for any Machine Learning library in Scala (I do not use Scala, so I cannot help you with suggestions in that regard)

$\endgroup$
1
$\begingroup$

Broadcasting your data and learn on it with different learning parameters per spark partition is a solution only if your data isn t so big it can fit in each machine memory. If you desire apply ML models at scale you have to deal with subquadratic complexity, if not you will to increase number of nodes accordingly to the complexity, but its something difficult from quadratic to bigger complexity. With Clustering4Ever we try to propose scalable clustering algorithms in scala and scala/spark. We will soon add some new algorithms that you may enjoy. Don t hesitate to ask for specific algorithms or add those you already implemented.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.