Binary text classification with TfidfVectorizer gives ValueError: setting an array element with a sequence

Question

I am using pandas and scikti-learn to do binary text classification using text features encoded using TfidfVectorizer on a DataFrame. Here is some dummy code that illustrates what I'm doing:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
data_dict = {'tid': [0,1,2,3,4,5,6,7,8,9],
         'text':['This is the first.', 'This is the second.', 'This is the third.', 'This is the fourth.', 'This is the fourth.', 'This is the fourth.', 'This is the nintieth.', 'This is the fourth.', 'This is the fourth.', 'This is the first.'],
         'cat':[0,0,1,1,1,1,1,0,0,0]}
df = pd.DataFrame(data_dict)
tfidf = TfidfVectorizer(analyzer='word')
df['text'] = tfidf.fit_transform(df['text'])
X_train, X_test, y_train, y_test = train_test_split(df[['tid', 'text']], df[['cat']])
clf = LinearSVC()
clf.fit(X_train, y_train)

This gives the following error:

Traceback (most recent call last):

  File "<ipython-input-151-b0953fbb1d6e>", line 1, in <module>
    clf.fit(X, y)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\svm\classes.py", line 227, in fit
    dtype=np.float64, order="C")

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 573, in check_X_y
    ensure_min_features, warn_on_dtype, estimator)

  File "C:\Users\Me\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 433, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)

ValueError: setting an array element with a sequence.

I have found numerous posts (e.g. here, here) mentioning that this error can indicate non-uniformity of the data. This post for the same error suggests it can also be due to a data typing issue. However, I can't see how my very simple example could be due to either of these. There is surely something simple I am missing. Help!

Answer 1

TfidfVectorizer returns a (sparse) 2-D array or a matrix. You can't set the column df['text'] to a matrix without messing up the dimensions. Instead, you need to concatenate the result from TfidfVectorizer with the remaining features in the dataframe.

df_text = pd.DataFrame(tfidf.fit_transform(df['text']).toarray()) 
X_train, X_test, y_train, y_test = train_test_split(pd.concat([df[['tid']],df_text],axis=1), df[['cat']])

Answer 2

One possible issue: train_test_split is expecting to return four values: X_train, X_test, y_train, y_test. It would be more likely to work if you used, in the next line, clf.fit(X_train, y_train). I think your toy data needs to have more in it to make train_test_split work intelligently: if I make the changes above, I get a ValueError: not enough values to unpack (expected 4, got 2).

So, try using only the training data in the fit routine, and try expanding out the toy data set to have more values.