I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark?

Isn't the whole purpose of running scripts on PySpark to overcome the limitations of packages like Pandas?

Does Pandas performance improve if it run on Spark? Is it compatible with Spark's RDD ?


There is no need for pandas module to be installed because your data is generally stored in spark RDD or spark dataframes objects.

The only interest I have found using Spark with pandas is when you want to load a local CSV / Excel dataset and then transform it into a spark dataframe. The "createDataFrame" method handles this approach.

>>> sqlContext.createDataFrame(pandas.DataFrame([[1, 2]])).collect()  
[Row(0=1, 1=2)]

Actually yes! While PySpark's built-in data frames are optimized for large datasets, they actually performs worse (i.e. slower) on small datasets, typically less than 500gb. Running Pandas in Spark can be very useful if you are working with a different sizes of datasets, some of which are small and can be held on a local machine. That being said, if you have data that is too big to fit on one machine, you will need to use spark data frames.

Here are some good links to learn more about Pandas for Spark:

  1. stack overflow question

  2. video explanation

  3. differences between pandas & spark data frames


There is a project that you may be interested in called Koalas which allows running pandas on top of Spark.

Koalas: pandas API on Apache Spark The Koalas project makes data scientists more productive when interacting with big data, by

implementing the pandas DataFrame API on top of Apache Spark. pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:

Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.

Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).

The most obvious benefit here being the last point made:

Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.