I have been working with Python for machine learning and have a fair amount of code written in Python using libraries such as scikit-learn, pandas, and numpy. Recently, I’ve been faced with larger datasets that require distributed computing to handle efficiently.

I understand that Apache Spark and PySpark are often used for this purpose. However, I am looking for a way to leverage the power of Spark without having to rewrite my existing Python code in PySpark or learn PySpark from scratch.

Is there an interface or library that allows me to run my existing Python ML code on a Spark cluster? Ideally, this would allow me to take advantage of distributed computing without needing to significantly modify my code or learn a new library in depth.

Any suggestions or guidance would be greatly appreciated. Thank you!


1 Answer 1


There's a library called joblib-spark that you can use to leverage a Spark cluster. It lets you take the Scikit-learn code you've written and train it in a distributed, parallel way across a Spark cluster which helps if you're if you're training large models on medium-sized datasets that can fit in the memory of a single node. You could also use Docker and Kubernetes which would allow you to handle large data in a distributed environment.

  1. https://github.com/joblib/joblib-spark,
  2. https://docs.docker.com/
  3. https://kubernetes.io/docs/home/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.