0
$\begingroup$

I have two RDDs named "data" and "model", they are repartitioned by key described as below : enter image description here

Does the tuple records with the same key reside in the same node in my cluster ?

Should it save IO cost in shuffle operation, such as "data.cogroup(model)" , if it comes true ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The tuple of one partition is always on the same node because a partition itself is impartible. So if you do a groupBy or write your own partitioner which partitions by key, all records with the same key/partition number will be shuffled to the same node.

Otherwise, transformations like mapPartition which pass an iterator to a user defined function wouldn't work.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.