0
$\begingroup$

Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$

I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e.

$result=[(0, 2), (0, 3), (1, 0)]$

I tried taking cartesian but it is giving other possibilities.
Also I tried with reverse of the above and then joining but it is also giving many use cases.
Please let me know the correct approach to solve the above.

|improve this question
$\endgroup$
  • $\begingroup$ I meant the logic behind this that shouldn't be like Cartesian product. I'm gonna delete my answer to make your post up. $\endgroup$ – Fatemeh Asgarinejad Aug 17 '19 at 19:48
  • $\begingroup$ Cartisian products gives you all the possible products while what you are aiming to get is based on transitivity among tuples in a set. This code can be a simple also time/energy consuming approach when there are many samples for this problem, but the logic behind is true. Time: 𝑂(𝑛2) O ( n 2 ) result = [] for i in A: for j in A: if i[1] == j[0]: print(i[0],j[1]) result.append((i[0],j[1])) There should be better ways, I will refresh the post If I find. $\endgroup$ – Fatemeh Asgarinejad Aug 17 '19 at 19:48
  • $\begingroup$ @FatemehAsgarinejad I got the answer by taking reverse of the above list and then using inner join of rdd to get all corresponding transitive pairs. After removing duplicates i.e. (b,a) and same edges (a,a) or (b,b) got the resulting rdd. Thanks for the help. $\endgroup$ – ten do Aug 18 '19 at 17:55
1
$\begingroup$

I got the answer by taking reverse of the above list and then using inner join of rdd to get all corresponding transitive pairs. After removing duplicates i.e. (b,a) and same edges (a,a) or (b,b) got the resulting rdd.

Thanks for the help.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.