8
$\begingroup$

What happens when we do repartition on a PySpark dataframe based on the column. For example

dataframe.repartition('id')

Does this moves the data with the similar 'id' to the same partition? How does the spark.sql.shuffle.partitions value affect the repartition?

| improve this question | | | | |
$\endgroup$
8
$\begingroup$

The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations.

dataframe.repartition('id') creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's, you'll end up with inconsistently sized partitions.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.