I have the dataframe which has two colums(Reviews and Label):

  Reviews                                     Label  
0 [cameron, diaz, woman, marri, judg, play]   1  
1 [turgid, dialogu, feebl, character, harvey] 0  
2 [misfortun, watch, rubbish, sky, cinema]    1 

I want to apply the TfidfVectorizer on the DF.
I have written the following code.

from sklearn.feature_extraction.text import TfidfVectorizer  
cv = TfidfVectorizer()   
df_xcv = cv.fit_transform(df_x)  

which is giving an error:

AttributeError: 'list' object has no attribute 'lower'

Why is this throwing an error?


Your reviews column is a column of lists, and not text. Tfidf Vectorizer works on text. I see that your reviews column is just a list of relevant polarity defining adjectives. A simple workaround is:

df['Reviews']=[" ".join(review) for review in df['Reviews'].values]

And then run the vectorizer again. That will fix the problem.

  • $\begingroup$ thanks its working fine. but now its giving an error array is too big; as my data set has 25000 records. how i can remove this error with out compromising on data frame $\endgroup$ Nov 6 '17 at 14:13
  • $\begingroup$ Paste the traceback. $\endgroup$ Nov 6 '17 at 14:26
  • $\begingroup$ @ Himanshu Rai ValueError Traceback (most recent call last) <ipython-input-4-4f3a2ce0230c> in <module>() 5 cv = TfidfVectorizer() 6 df_xcv = cv.fit_transform(df_x) ----> 7 a=df_xcv.toarray() 8 cv.get_feature_names() $\endgroup$ Nov 6 '17 at 14:45
  • $\begingroup$ Hard to read this. Which line is throwing error? $\endgroup$ Nov 6 '17 at 14:50
  • $\begingroup$ a=df_xcv.toarray() at this line. @Himanshu Rai $\endgroup$ Nov 6 '17 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.