I am new to the field of machine learning and recently learned the basics and working out various algorithms in python using libraries such as pandas, numpy, matplotlib, scikitlearn , etc. I started learning about working Bigdata by distributing it and using Apache Spark's library MLlib to load and apply algorithms on it. So is working with MLlib the only way on Spark or is there any other way to use pandas and other libraries on distributed data?


Yes. There are a million ways to engineer a solution.

You can use pandas for data wrangling and distribute this task via parallel python.

I personally use python for distributed - data acquisition, data wrangling, and algorithmic computation. This way i get the capability of Multi-machine, multi CPU, & multi GPU.

If you want to use python all they way, i recommend recording all the different options in terms of libraries, then order them by most active in development releases, or by whatever criterion you value the most.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.