5
$\begingroup$

I have the dataframe which has two colums(Reviews and Label):

  Reviews                                     Label  
0 [cameron, diaz, woman, marri, judg, play]   1  
1 [turgid, dialogu, feebl, character, harvey] 0  
2 [misfortun, watch, rubbish, sky, cinema]    1 

I want to apply the TfidfVectorizer on the DF.
I have written the following code.

from sklearn.feature_extraction.text import TfidfVectorizer  
df_x=train_df["Reviews"]  
df_y=train_df["Label"]  
cv = TfidfVectorizer()   
df_xcv = cv.fit_transform(df_x)  
a=df_xcv.toarray()  
cv.get_feature_names()  

which is giving an error:

AttributeError: 'list' object has no attribute 'lower'

Why is this throwing an error?

$\endgroup$
3
$\begingroup$

Your reviews column is a column of lists, and not text. Tfidf Vectorizer works on text. I see that your reviews column is just a list of relevant polarity defining adjectives. A simple workaround is:

df['Reviews']=[" ".join(review) for review in df['Reviews'].values]

And then run the vectorizer again. That will fix the problem.

$\endgroup$
  • $\begingroup$ thanks its working fine. but now its giving an error array is too big; as my data set has 25000 records. how i can remove this error with out compromising on data frame $\endgroup$ – Dhanshree Bagal Nov 6 '17 at 14:13
  • $\begingroup$ Paste the traceback. $\endgroup$ – Himanshu Rai Nov 6 '17 at 14:26
  • $\begingroup$ @ Himanshu Rai ValueError Traceback (most recent call last) <ipython-input-4-4f3a2ce0230c> in <module>() 5 cv = TfidfVectorizer() 6 df_xcv = cv.fit_transform(df_x) ----> 7 a=df_xcv.toarray() 8 cv.get_feature_names() $\endgroup$ – Dhanshree Bagal Nov 6 '17 at 14:45
  • $\begingroup$ Hard to read this. Which line is throwing error? $\endgroup$ – Himanshu Rai Nov 6 '17 at 14:50
  • $\begingroup$ a=df_xcv.toarray() at this line. @Himanshu Rai $\endgroup$ – Dhanshree Bagal Nov 6 '17 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.