1
$\begingroup$

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.

So for i.e. [8,7,6,7,8,8,5]

How can I manipulate the RDD so it only has Monday, Wednesday, Friday values? There are no column names by the way. But the PySpark platform seems to have _co1,_co2,...,_coN as columns.

$\endgroup$
4
$\begingroup$

I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer spark 2.3.2)

First convert rdd to DataFrame:

df = rdd.toDF(["M","Tu","W","Th","F","Sa","Su"])

Then select days you want to work with:

df.select("M","W","F").show(3)

Or directly use map with lambda:

rdd.map(lambda x: [x[i] for i in [0,2,4])

Hope it helps!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.