testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)

I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute 'select'


You could try the following,

testPassengerID = test.select('PassengerID').rdd

this would select the column PassengerID and convert it into a rdd

  • $\begingroup$ One issue with this is that you get a row back out and so then might have to do what @wabbit suggests. $\endgroup$ – groceryheist Mar 13 '19 at 20:31

'RDD' object has no attribute 'select'

This means that test is in fact an RDD and not a dataframe (which you are assuming it to be). Either you convert it to a dataframe and then apply select or do a map operation over the RDD.

Please let me know if you need any help around this.

  • $\begingroup$ Is it possible to select multiple columns? $\endgroup$ – Kent Wong Nov 27 '19 at 3:53
  • $\begingroup$ Yes it is :) You could use df.select(*list_of_columns_to_select) $\endgroup$ – Shagun Sodhani Nov 27 '19 at 12:06
  • $\begingroup$ So you must use a data frame then? Not possible with just a RDD then? $\endgroup$ – Kent Wong Nov 27 '19 at 16:49
  • $\begingroup$ I think there os no way but my knowledge of RDDs is rustic now :) $\endgroup$ – Shagun Sodhani Nov 27 '19 at 22:58

Assuming you have an RDD each row of which is of the form (passenger_ID, passenger_name), you can do rdd.map(lambda x: x[0]). This is for a basic RDD

If you use Spark sqlcontext there are functions to select by column name.


If your RDD happens to be in the form of a dictionary, this is how it can be done using PySpark:

Define the fields you want to keep in here:

field_list =[]

Create a function to keep specific keys within a dict input

def f(x):
    d = {}
    for k in x:
        if k in field_list:
            d[k] = x[k]
    return d

And just map after that, with x being an RDD row

rdd_subset = rdd.map(lambda x: f(x))


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.