5
$\begingroup$

I am using pandas and scikti-learn to do binary text classification using text features encoded using TfidfVectorizer on a DataFrame. Here is some dummy code that illustrates what I'm doing:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
data_dict = {'tid': [0,1,2,3,4,5,6,7,8,9],
         'text':['This is the first.', 'This is the second.', 'This is the third.', 'This is the fourth.', 'This is the fourth.', 'This is the fourth.', 'This is the nintieth.', 'This is the fourth.', 'This is the fourth.', 'This is the first.'],
         'cat':[0,0,1,1,1,1,1,0,0,0]}
df = pd.DataFrame(data_dict)
tfidf = TfidfVectorizer(analyzer='word')
df['text'] = tfidf.fit_transform(df['text'])
X_train, X_test, y_train, y_test = train_test_split(df[['tid', 'text']], df[['cat']])
clf = LinearSVC()
clf.fit(X_train, y_train)

This gives the following error:

Traceback (most recent call last):

  File "<ipython-input-151-b0953fbb1d6e>", line 1, in <module>
    clf.fit(X, y)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\svm\classes.py", line 227, in fit
    dtype=np.float64, order="C")

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 573, in check_X_y
    ensure_min_features, warn_on_dtype, estimator)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 433, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)

ValueError: setting an array element with a sequence.

I have found numerous posts (e.g. here, here) mentioning that this error can indicate non-uniformity of the data. This post for the same error suggests it can also be due to a data typing issue. However, I can't see how my very simple example could be due to either of these. There is surely something simple I am missing. Help!

$\endgroup$
2
$\begingroup$

TfidfVectorizer returns a (sparse) 2-D array or a matrix. You can't set the column df['text'] to a matrix without messing up the dimensions. Instead, you need to concatenate the result from TfidfVectorizer with the remaining features in the dataframe.

df_text = pd.DataFrame(tfidf.fit_transform(df['text']).toarray()) 
X_train, X_test, y_train, y_test = train_test_split(pd.concat([df[['tid']],df_text],axis=1), df[['cat']])
| improve this answer | |
$\endgroup$
  • $\begingroup$ Would you mind providing example code (or link to a relevant solution) to do this please? $\endgroup$ – ongenz Sep 14 '18 at 17:58
  • $\begingroup$ ok see the edit $\endgroup$ – oW_ Sep 14 '18 at 18:10
  • $\begingroup$ Thanks a lot! That works pretty well, although I still get a DataConversionWarning - is this likely to be problematic or can I get by with ignoring it? $\endgroup$ – ongenz Sep 14 '18 at 18:22
  • $\begingroup$ Unfortunately, although this solution is helpful, it does not work on my real dataset, which is much larger and results in a MemoryError. I need to keep the data in sparse format (i.e. the format output by the vectorizer). Do you know how to do that? $\endgroup$ – ongenz Sep 14 '18 at 18:45
  • 1
    $\begingroup$ You can do this using FeatureUnion where one transformer does the tfidf transform and another one that selects the non-text features or by working directly with the scipy sparse matrices instead of pandas dataframes $\endgroup$ – oW_ Sep 14 '18 at 19:58
1
$\begingroup$

One possible issue: train_test_split is expecting to return four values: X_train, X_test, y_train, y_test. It would be more likely to work if you used, in the next line, clf.fit(X_train, y_train). I think your toy data needs to have more in it to make train_test_split work intelligently: if I make the changes above, I get a ValueError: not enough values to unpack (expected 4, got 2).

So, try using only the training data in the fit routine, and try expanding out the toy data set to have more values.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for pointing this out. I have updated my post accordingly. Unfortunately, the error persists... $\endgroup$ – ongenz Sep 14 '18 at 17:45
  • $\begingroup$ Where's the train_test_split call? $\endgroup$ – Adrian Keister Sep 14 '18 at 17:51
  • $\begingroup$ Oops, sorry, copy-paste error. I've put it back. $\endgroup$ – ongenz Sep 14 '18 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.