6
$\begingroup$

I have the dataframe which has two colums(Reviews and Label): 

  Reviews                                     Label  
0 [cameron, diaz, woman, marri, judg, play]   1  
1 [turgid, dialogu, feebl, character, harvey] 0  
2 [misfortun, watch, rubbish, sky, cinema]    1

I want to apply the TfidfVectorizer on the DF.
I have written the following code. 

from sklearn.feature_extraction.text import TfidfVectorizer  
df_x=train_df["Reviews"]  
df_y=train_df["Label"]  
cv = TfidfVectorizer()   
df_xcv = cv.fit_transform(df_x)  
a=df_xcv.toarray()  
cv.get_feature_names()

which is giving an error:

AttributeError: 'list' object has no attribute 'lower'

Why is this throwing an error?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Your reviews column is a column of lists, and not text. Tfidf Vectorizer works on text. I see that your reviews column is just a list of relevant polarity defining adjectives. A simple workaround is:

df['Reviews']=[" ".join(review) for review in df['Reviews'].values]

And then run the vectorizer again. That will fix the problem.

Improve this answer
$\endgroup$
12
  • $\begingroup$ thanks its working fine. but now its giving an error array is too big; as my data set has 25000 records. how i can remove this error with out compromising on data frame $\endgroup$
    – Dhanshree Bagal
    Commented Nov 6, 2017 at 14:13
  • $\begingroup$ Paste the traceback. $\endgroup$
    – Himanshu Rai
    Commented Nov 6, 2017 at 14:26
  • $\begingroup$ @ Himanshu Rai ValueError Traceback (most recent call last) <ipython-input-4-4f3a2ce0230c> in <module>() 5 cv = TfidfVectorizer() 6 df_xcv = cv.fit_transform(df_x) ----> 7 a=df_xcv.toarray() 8 cv.get_feature_names() $\endgroup$
    – Dhanshree Bagal
    Commented Nov 6, 2017 at 14:45
  • $\begingroup$ Hard to read this. Which line is throwing error? $\endgroup$
    – Himanshu Rai
    Commented Nov 6, 2017 at 14:50
  • $\begingroup$ a=df_xcv.toarray() at this line. @Himanshu Rai $\endgroup$
    – Dhanshree Bagal
    Commented Nov 6, 2017 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.