testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)
I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute 'select'
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ You can access columns pandas-style using DataFrame notation. $\endgroup$– EmreJan 3, 2016 at 4:34
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
You could try the following,
testPassengerID = test.select('PassengerID').rdd
this would select the column PassengerID and convert it into a rdd
-
$\begingroup$ One issue with this is that you get a row back out and so then might have to do what @wabbit suggests. $\endgroup$ Mar 13, 2019 at 20:31
$\begingroup$
$\endgroup$
4
'RDD' object has no attribute 'select'
This means that test is in fact an RDD and not a dataframe (which you are assuming it to be). Either you convert it to a dataframe and then apply select or do a map operation over the RDD.
Please let me know if you need any help around this.
-
$\begingroup$ Is it possible to select multiple columns? $\endgroup$ Nov 27, 2019 at 3:53
-
$\begingroup$ Yes it is :) You could use
df.select(*list_of_columns_to_select)$\endgroup$ Nov 27, 2019 at 12:06 -
$\begingroup$ So you must use a data frame then? Not possible with just a RDD then? $\endgroup$ Nov 27, 2019 at 16:49
-
$\begingroup$ I think there os no way but my knowledge of RDDs is rustic now :) $\endgroup$ Nov 27, 2019 at 22:58
$\begingroup$
$\endgroup$
Assuming you have an RDD each row of which is of the form (passenger_ID, passenger_name), you can do rdd.map(lambda x: x[0]). This is for a basic RDD
If you use Spark sqlcontext there are functions to select by column name.
$\begingroup$
$\endgroup$
If your RDD happens to be in the form of a dictionary, this is how it can be done using PySpark:
Define the fields you want to keep in here:
field_list =[]
Create a function to keep specific keys within a dict input
def f(x):
d = {}
for k in x:
if k in field_list:
d[k] = x[k]
return d
And just map after that, with x being an RDD row
rdd_subset = rdd.map(lambda x: f(x))
