# Python Sklearn TfidfVectorizer Feature not matching; delete?

I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.

The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error:

ValueError: X has 4877 features per sample; expecting 2799


Is there a way to delete any features in x_test that were not used in x_train?

I know if I had used a countverctorizer, i could have bypassed the error by not using fit_transform on x_test. But since it is TfidfVectorizer, it won't let me bypass.

I also tried imputation but couldn't get it to work.

Thanks

You should not experience this problem if you use TfidfVectorizer properly.

Demo:

In [58]: from sklearn.feature_extraction.text import TfidfVectorizer


source text

In [59]: text = """I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.
...:
...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error"""


let's tokenize it to a list of sentenses:

In [60]: from nltk import  sent_tokenize

In [61]: vect = TfidfVectorizer()

In [62]: data = sent_tokenize(text)


yields:

In [63]: data
Out[63]:
['I trained a classifier using TfidfVectorizer in Sklearn.',
'I then pickled the model for future use.',
'The new x_test that I want to make predictions on, has more features than the x_train from the model.',
'This is the resulting error']


now we can fit and transform our data set:

In [64]: X = vect.fit_transform(data)


result:

In [65]: X
Out[65]:
<4x31 sparse matrix of type '<class 'numpy.float64'>'
with 34 stored elements in Compressed Sparse Row format>

In [66]: vect.get_feature_names()
Out[66]:
['classifier',
'error',
'features',
'for',
'from',
'future',
'has',
'in',
'is',
'make',
'model',
'more',
'new',
'on',
'pickled',
'predictions',
'resulting',
'sklearn',
'tfidfvectorizer',
'than',
'that',
'the',
'then',
'this',
'to',
'trained',
'use',
'using',
'want',
'x_test',
'x_train']


now let's feed it a data set with unknown words (features):

In [67]: new_dataset = ["let's see what happens to unknown words", "Yet another sentence."]

In [68]: X2 = vect.transform(new_dataset)

In [69]: X2
Out[69]:
<2x31 sparse matrix of type '<class 'numpy.float64'>'
with 1 stored elements in Compressed Sparse Row format>


it worked properly - all unknown features (words) have been ignored:

In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)
Out[70]:
classifier  error  features  for  from  future  has   in   is  make   ...     the  then  this   to  trained  use  using  want  \
0         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  1.0      0.0  0.0    0.0   0.0
1         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  0.0      0.0  0.0    0.0   0.0

x_test  x_train
0     0.0      0.0
1     0.0      0.0

[2 rows x 31 columns]

• Thanks for this comprehensive reply. Ive been trying for hours to solve an issue i'm having using your code. Sent_tokenize is giving me an error: expected string or bytes-like object; i tried using nltk word_tokenizer. But couldn't figure it out. I'm very new at this. Can i share my full code with you to have a look? And how would i be able to share? I don't code area like they have on stackoverflow. – Yousuf Apr 22 '18 at 16:20
• @Yousuf, the best option would be to prepare a small reproducible data set and post it somewhere (GitHub, any file-exchange web site) and ask a new question (including your code). But the most important thing is a small reproducible data set and your desired data set... – MaxU Apr 22 '18 at 18:32
• Doing the same thing but I wanted to try something different. I don't want unknown features to be ignored. I want them to participate in prediction. Any idea from where I can start. – Hammad Hassan Nov 1 '18 at 8:14

I think you would need to delete exactly those features (columns) that are not known to your model.

Imagine that you trained your model on the following features:

• human weight
• human height
• sex
• age

and now you want to feed it a test data set containing the following features:

• human weight
• human height
• sex
• age
• eye color
• name
• zip code

so if if you randomly reduce it to the correct shape and get:

• eye color (should be: human weight)
• human height
• zip code (should be: sex)
• name (should be: age)

what kind of accuracy would you expect in this case?

• Thanks for the answer. That is true. So do i have to manually delete those from my file? Or is there a function ? – Yousuf Apr 21 '18 at 18:15
• @Yousuf, well it depends on how do you process your data... If you know indices of needed features, then it's fairly easy to implement in Numpy or Pandas ... – MaxU Apr 21 '18 at 18:17