Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark?
Isn't the whole purpose of running scripts on PySpark to overcome the limitations of packages like Pandas?
Does Pandas performance improve if it run on Spark? Is it compatible with Spark's RDD ?
There is no need for pandas module to be installed because your data is generally stored in spark RDD or spark dataframes objects.
The only interest I have found using Spark with pandas is when you want to load a local CSV / Excel dataset and then transform it into a spark dataframe. The "createDataFrame" method handles this approach.
>>> sqlContext.createDataFrame(pandas.DataFrame([[1, 2]])).collect()
[Row(0=1, 1=2)]
Actually yes! While PySpark's built-in data frames are optimized for large datasets, they actually performs worse (i.e. slower) on small datasets, typically less than 500gb. Running Pandas in Spark can be very useful if you are working with a different sizes of datasets, some of which are small and can be held on a local machine. That being said, if you have data that is too big to fit on one machine, you will need to use spark data frames.
Here are some good links to learn more about Pandas for Spark:
There is a project that you may be interested in called Koalas which allows running pandas on top of Spark.
Koalas: pandas API on Apache Spark The Koalas project makes data scientists more productive when interacting with big data, by
implementing the pandas DataFrame API on top of Apache Spark. pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:
Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).
The most obvious benefit here being the last point made:
Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).
