5
$\begingroup$

I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The reason I am confused is 1)The scikit learn python codes don't scale in big data. 2)The spark mlib is written in scala and not in python. 3)Although h2o is open sourced I don't think it is written in python.

$\endgroup$
1
1
$\begingroup$

Spark does have a pretty good Python API, check out this tutorial.

For traditional Hadoop stack, take a look at mrjob, it lets you write MapReduce jobs in Python and run them on several platforms.

$\endgroup$
0
1
$\begingroup$

While checking on python xgboost I found the existence of this open source project that helps create scalable machine learning program.Should be worth exploring.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.