How to select multiple columns in a RDD with Spark (pySpark)?

Question

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.

So for i.e. [8,7,6,7,8,8,5]

How can I manipulate the RDD so it only has Monday, Wednesday, Friday values? There are no column names by the way. But the PySpark platform seems to have _co1,_co2,...,_coN as columns.

Answer 1

I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer spark 2.3.2)

First convert rdd to DataFrame:

df = rdd.toDF(["M","Tu","W","Th","F","Sa","Su"])

Then select days you want to work with:

df.select("M","W","F").show(3)

Or directly use map with lambda:

rdd.map(lambda x: [x[i] for i in [0,2,4])

Hope it helps!