11
$\begingroup$

What happens when we do repartition on a PySpark dataframe based on the column. For example

dataframe.repartition('id')

Does this moves the data with the similar 'id' to the same partition? How does the spark.sql.shuffle.partitions value affect the repartition?

$\endgroup$

1 Answer 1

12
$\begingroup$

The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations.

dataframe.repartition('id') creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's, you'll end up with inconsistently sized partitions.

$\endgroup$
1
  • $\begingroup$ Ideally means that all the data of a particular ID will be located on the same worker node right? $\endgroup$ Nov 20, 2020 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.