Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I’m using Spark Scala for multiclass classification, and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, Multinomial and Bernoulli are supported, where I would need to use Gaussian, and LogisticRegressionWithLBFGS would not be suitable either.
I know that in Python, you can integrate sci-kit learn with Spark, but are there any options when using Scala?
Interested to hear people’s thoughts on this.
The way you can use scikit-learn is basically broadcasting your data to the workers and then do different folds of crossvalidation or different parameter settings in your grid_search on different workers. That is all that the scikit-learn package in pySpark does as far as I know. This is similar to a normal mapping. Implementing this should be relatively easy for any Machine Learning library in Scala (I do not use Scala, so I cannot help you with suggestions in that regard)
Broadcasting your data and learn on it with different learning parameters per spark partition is a solution only if your data isn t so big it can fit in each machine memory. If you desire apply ML models at scale you have to deal with subquadratic complexity, if not you will to increase number of nodes accordingly to the complexity, but its something difficult from quadratic to bigger complexity. With Clustering4Ever we try to propose scalable clustering algorithms in scala and scala/spark. We will soon add some new algorithms that you may enjoy. Don t hesitate to ask for specific algorithms or add those you already implemented.
