Python Sklearn TfidfVectorizer Feature not matching; delete?

Question

I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.

The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error:

ValueError: X has 4877 features per sample; expecting 2799

Is there a way to delete any features in x_test that were not used in x_train?

I know if I had used a countverctorizer, i could have bypassed the error by not using fit_transform on x_test. But since it is TfidfVectorizer, it won't let me bypass.

I also tried imputation but couldn't get it to work.

Thanks

Answer 1

You should not experience this problem if you use TfidfVectorizer properly.

Demo:

In [58]: from sklearn.feature_extraction.text import TfidfVectorizer

source text

In [59]: text = """I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.
    ...:
    ...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error"""

let's tokenize it to a list of sentenses:

In [60]: from nltk import  sent_tokenize

In [61]: vect = TfidfVectorizer()

In [62]: data = sent_tokenize(text)

yields:

In [63]: data
Out[63]:
['I trained a classifier using TfidfVectorizer in Sklearn.',
 'I then pickled the model for future use.',
 'The new x_test that I want to make predictions on, has more features than the x_train from the model.',
 'This is the resulting error']

now we can fit and transform our data set:

In [64]: X = vect.fit_transform(data)

result:

In [65]: X
Out[65]:
<4x31 sparse matrix of type '<class 'numpy.float64'>'
        with 34 stored elements in Compressed Sparse Row format>

In [66]: vect.get_feature_names()
Out[66]:
['classifier',
 'error',
 'features',
 'for',
 'from',
 'future',
 'has',
 'in',
 'is',
 'make',
 'model',
 'more',
 'new',
 'on',
 'pickled',
 'predictions',
 'resulting',
 'sklearn',
 'tfidfvectorizer',
 'than',
 'that',
 'the',
 'then',
 'this',
 'to',
 'trained',
 'use',
 'using',
 'want',
 'x_test',
 'x_train']

now let's feed it a data set with unknown words (features):

In [67]: new_dataset = ["let's see what happens to unknown words", "Yet another sentence."]

In [68]: X2 = vect.transform(new_dataset)

In [69]: X2
Out[69]:
<2x31 sparse matrix of type '<class 'numpy.float64'>'
        with 1 stored elements in Compressed Sparse Row format>

it worked properly - all unknown features (words) have been ignored:

In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)
Out[70]:
   classifier  error  features  for  from  future  has   in   is  make   ...     the  then  this   to  trained  use  using  want  \
0         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  1.0      0.0  0.0    0.0   0.0
1         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  0.0      0.0  0.0    0.0   0.0

   x_test  x_train
0     0.0      0.0
1     0.0      0.0

[2 rows x 31 columns]

Answer 2

I think you would need to delete exactly those features (columns) that are not known to your model.

Imagine that you trained your model on the following features:

• human weight
• human height
• sex
• age

and now you want to feed it a test data set containing the following features:

• human weight
• human height
• sex
• age
• eye color
• name
• zip code

so if if you randomly reduce it to the correct shape and get:

• eye color (should be: human weight)
• human height
• zip code (should be: sex)
• name (should be: age)

what kind of accuracy would you expect in this case?