5
$\begingroup$

I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The reason I am confused is 1)The scikit learn python codes don't scale in big data. 2)The spark mlib is written in scala and not in python. 3)Although h2o is open sourced I don't think it is written in python.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
1
$\begingroup$

Spark does have a pretty good Python API, check out this tutorial.

For traditional Hadoop stack, take a look at mrjob, it lets you write MapReduce jobs in Python and run them on several platforms.

Improve this answer
$\endgroup$
0
1
$\begingroup$

While checking on python xgboost I found the existence of this open source project that helps create scalable machine learning program.Should be worth exploring.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.