Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.

So for i.e. [8,7,6,7,8,8,5]

How can I manipulate the RDD so it only has Monday, Wednesday, Friday values? There are no column names by the way. But the PySpark platform seems to have _co1,_co2,...,_coN as columns.


1 Answer 1


I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer spark 2.3.2)

First convert rdd to DataFrame:

df = rdd.toDF(["M","Tu","W","Th","F","Sa","Su"])

Then select days you want to work with:


Or directly use map with lambda:

rdd.map(lambda x: [x[i] for i in [0,2,4])

Hope it helps!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.