# Sklearn tfidf vectorize returns different shape after fit_transform()

I'm new to ML and trying out basic samples using sklearn. I have achieved converting text (single dimension) to numbers using TF-IDF and got the predictions correct.

Now I have a different use-case to predict Work Hours based on Marital-Status and Relationship Status so my training data is now two dimension.

I have DataFrame total of 24420 rows * 2 columns of shape --> (24420, 2)

After this DataFrame is been passed to TfidfVectorizer.fit_transform() function it turns to shape --> (2, 3) Not sure why!

And while training mnb.fit(x_train_tf, y_train) it returns an error

ValueError: Found input variables with inconsistent numbers of samples: [2, 24420]

You could see the initial DataFrame shape of 24420*2 it transposed to 2*24420

All code is up here.

Questions:

1. Why after fit_transform() the shape changes to (2,3) what happened to (24420,2) data? What does this denotes?

2. Why does the ValueError: Found input variables with inconsistent numbers of samples [2, 24420] occurs?

3. Why initial DataFrame shape changed to 2*24420?

What is wrong with the implementation?

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this -

>>> x_train_tf.toarray()
array([[ 0.70710678,  0.        ,  0.70710678],
[ 0.        ,  1.        ,  0.        ]])


If you check what features you are getting, you will see -

>>> tfidf.get_feature_names()
[u'marital', u'relationship', u'status']


What's happening is, while passing dataframe, the TfidfVectorizer is only taking the column names and converting them into numeric form.

I don't think you need to use tfidf here. As far as I understand, your data is categorical text, so use pandas.get_dummies() instead of tfidf. This will convert your categorical data to numeric form which you can use for modelling.

>>> pd.get_dummies(x_train.iloc[:10,])
marital-status_Divorced  marital-status_Married-civ-spouse  marital-status_Never-married  marital-status_Widowed  relationship_Husband  relationship_Not-in-family  relationship_Own-child  relationship_Unmarried
1521                   1                          0                              0                        0                       0                            1                         0                        0
2274                   0                          1                              0                        0                       1                            0                         0                        0
20209                  0                          1                              0                        0                       1                            0                         0                        0
5529                   0                          1                              0                        0                       1                            0                         0                        0
27639                  0                          1                              0                        0                       1                            0                         0                        0
26670                  0                          0                              1                        0                       0                            1                         0                        0
16635                  0                          0                              0                        1                       0                            0                         0                        1
30824                  0                          0                              1                        0                       0                            1                         0                        0
15181                  0                          0                              1                        0                       0                            0                         1                        0
9119                   1                          0                              0                        0                       0                            0                         1                        0


Here, for demo purpose, I have taken only 10 rows.

• thanks, my actual case includes data which are NOT categorical and have over 6 lakhs records. So I have to use TF-IDF with two column data. Do you know how to solve that? – yashhy Jan 16 '18 at 12:24

I had a similar problem. What I was doing was this: I loaded a pretrained tfidf vectorizer. And on the new data to be predicted, I called vectorizer.fit_transform() and I got similar error.

What solved the issue was calling vectorizer.transform(). It is because, fit_transform() will fit the current data in the model, which is not what we are seeking because vectorizer has already been fitted. We just need to transform the new data to model which has been created. So, calling vectorizer.transform() did the work.