8
$\begingroup$

testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)

I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute 'select'

$\endgroup$
1

4 Answers 4

4
$\begingroup$

You could try the following,

testPassengerID = test.select('PassengerID').rdd

this would select the column PassengerID and convert it into a rdd

$\endgroup$
1
  • $\begingroup$ One issue with this is that you get a row back out and so then might have to do what @wabbit suggests. $\endgroup$ Mar 13, 2019 at 20:31
3
$\begingroup$

'RDD' object has no attribute 'select'

This means that test is in fact an RDD and not a dataframe (which you are assuming it to be). Either you convert it to a dataframe and then apply select or do a map operation over the RDD.

Please let me know if you need any help around this.

$\endgroup$
4
  • $\begingroup$ Is it possible to select multiple columns? $\endgroup$
    – Kent Wong
    Nov 27, 2019 at 3:53
  • $\begingroup$ Yes it is :) You could use df.select(*list_of_columns_to_select) $\endgroup$ Nov 27, 2019 at 12:06
  • $\begingroup$ So you must use a data frame then? Not possible with just a RDD then? $\endgroup$
    – Kent Wong
    Nov 27, 2019 at 16:49
  • $\begingroup$ I think there os no way but my knowledge of RDDs is rustic now :) $\endgroup$ Nov 27, 2019 at 22:58
3
$\begingroup$

Assuming you have an RDD each row of which is of the form (passenger_ID, passenger_name), you can do rdd.map(lambda x: x[0]). This is for a basic RDD

If you use Spark sqlcontext there are functions to select by column name.

$\endgroup$
0
$\begingroup$

If your RDD happens to be in the form of a dictionary, this is how it can be done using PySpark:

Define the fields you want to keep in here:

field_list =[]

Create a function to keep specific keys within a dict input

def f(x):
    d = {}
    for k in x:
        if k in field_list:
            d[k] = x[k]
    return d

And just map after that, with x being an RDD row

rdd_subset = rdd.map(lambda x: f(x))

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.