I believe sci kit learn is written in python,however that not scalable.Spark mlib or ml is scalabale but written in scala.I am looking for an ongoing effort where a machine learning library is being built in python (available in github or so) so that I can contribute to that.Is anyone aware of such effort.

  • $\begingroup$ Define scalable. What exact requirements do you have? Do you only care about feeding data into models to get predictions or is training them a concern as well? I also wouldn't make the assertion that sci-kit isn't scalable, it most certainly is for quite a few (if not most) cases. $\endgroup$ – David Jul 10 '15 at 23:52
  • $\begingroup$ @David well, some of the scikit learn models such as random forest can run in parallel and can use all the cores in a node,but can't extend beyond that.To me scalability means the capacity to run on multiple nodes.Also I think it's the training that matters as that consumes most memory(specially in the case of text mining). $\endgroup$ – 0xF Jul 11 '15 at 2:10
  • 1
    $\begingroup$ Ok, that's helpful information. Scalable typically refers to being able to handle a large amount of input data for making predictions, since training is generally a one time cost and not performed on that much data (relatively speaking). It seems like you're really interested in using a natively distributed system, which spark makes a lot of sense for. $\endgroup$ – David Jul 11 '15 at 16:04
  • $\begingroup$ While checking on python xgboost I found the existence of this open source project github.com/dmlc/rabit that helps create scalable machine learning program.Should be worth exploring. $\endgroup$ – 0xF Jul 14 '15 at 6:12

Is there a specific reason beside the fact that you would like to contribute? I am asking because there is always pyspark that you can use, the Spark python API that exposes the Spark programming model to Python.

For deep learning specifically, there are a lot of frameworks built on top of Theano -which is a python library for mathematical expressions involving multi-dimensional arrays-, like Lasagne, so they are able to use GPU for intense training. Getting an EC2 instance with GPU on AWS is always an option.

  • $\begingroup$ I'm mainly interested in the area of deep learning.My company can't put their data on EC2 due to security reason hence GPU isn't an option for me.I use h2o deep learning on clusters,however it lacks some feature I think,such as setting learning rate for different layers.Since it is written in Java and my java is bit rusty I was looking for any open source scalable deep learning written in python.Yes pyspark gives access to clusters,but even if I build something on it'll work on top of scala layer and not on native python code.Hence I am on a search for an ongoing python project. $\endgroup$ – 0xF Jul 10 '15 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.