If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be distributed across the cluster? Or do i have to use pyspark dataframe operations and pyspark ml packages for operations to be distributed?

  • $\begingroup$ If you find the answer suitable, you can use the little hidden tick under the Up/down vote to accept the answer. Welcome to community. $\endgroup$ – TwinPenguins Apr 3 '20 at 13:50
  • $\begingroup$ See also github.com/databricks/koalas which is a pandas work-alike API that implements the operations on Spark, without using Pyspark APIs $\endgroup$ – Sean Owen Apr 18 '20 at 3:51

Short answer: NO.

The moment you convert the spark dataframe into a pandas dataframe, all of the subsequent operations (pandas, ml etc.) will be run on a single-core as those algorithms and programs are written in native-python and doesn't support multi-core distributions. In a nutshell, someone has to rewrite the whole sklearn to again to be compatible with spark.

Said that there are some progress to bring e.g. to distribute scikit-learn on a Spark cluster. Some functionalities like the expensive GridSearch are rewritten in spark and you can use it together with native sklearn in databricks, see this post, or Joblib Apache Spark Backend (previously known as spark-sklearn). This blog post is also worth reading as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.