2
$\begingroup$

I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.

The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error:

ValueError: X has 4877 features per sample; expecting 2799

Is there a way to delete any features in x_test that were not used in x_train?

I know if I had used a countverctorizer, i could have bypassed the error by not using fit_transform on x_test. But since it is TfidfVectorizer, it won't let me bypass.

I also tried imputation but couldn't get it to work.

Thanks

$\endgroup$
2
$\begingroup$

You should not experience this problem if you use TfidfVectorizer properly.

Demo:

In [58]: from sklearn.feature_extraction.text import TfidfVectorizer

source text

In [59]: text = """I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.
    ...:
    ...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error"""

let's tokenize it to a list of sentenses:

In [60]: from nltk import  sent_tokenize

In [61]: vect = TfidfVectorizer()

In [62]: data = sent_tokenize(text)

yields:

In [63]: data
Out[63]:
['I trained a classifier using TfidfVectorizer in Sklearn.',
 'I then pickled the model for future use.',
 'The new x_test that I want to make predictions on, has more features than the x_train from the model.',
 'This is the resulting error']

now we can fit and transform our data set:

In [64]: X = vect.fit_transform(data)

result:

In [65]: X
Out[65]:
<4x31 sparse matrix of type '<class 'numpy.float64'>'
        with 34 stored elements in Compressed Sparse Row format>

In [66]: vect.get_feature_names()
Out[66]:
['classifier',
 'error',
 'features',
 'for',
 'from',
 'future',
 'has',
 'in',
 'is',
 'make',
 'model',
 'more',
 'new',
 'on',
 'pickled',
 'predictions',
 'resulting',
 'sklearn',
 'tfidfvectorizer',
 'than',
 'that',
 'the',
 'then',
 'this',
 'to',
 'trained',
 'use',
 'using',
 'want',
 'x_test',
 'x_train']

now let's feed it a data set with unknown words (features):

In [67]: new_dataset = ["let's see what happens to unknown words", "Yet another sentence."]

In [68]: X2 = vect.transform(new_dataset)

In [69]: X2
Out[69]:
<2x31 sparse matrix of type '<class 'numpy.float64'>'
        with 1 stored elements in Compressed Sparse Row format>

it worked properly - all unknown features (words) have been ignored:

In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)
Out[70]:
   classifier  error  features  for  from  future  has   in   is  make   ...     the  then  this   to  trained  use  using  want  \
0         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  1.0      0.0  0.0    0.0   0.0
1         0.0    0.0       0.0  0.0   0.0     0.0  0.0  0.0  0.0   0.0   ...     0.0   0.0   0.0  0.0      0.0  0.0    0.0   0.0

   x_test  x_train
0     0.0      0.0
1     0.0      0.0

[2 rows x 31 columns]
$\endgroup$
  • $\begingroup$ Thanks for this comprehensive reply. Ive been trying for hours to solve an issue i'm having using your code. Sent_tokenize is giving me an error: expected string or bytes-like object; i tried using nltk word_tokenizer. But couldn't figure it out. I'm very new at this. Can i share my full code with you to have a look? And how would i be able to share? I don't code area like they have on stackoverflow. $\endgroup$ – Yousuf Apr 22 '18 at 16:20
  • $\begingroup$ @Yousuf, please consider accepting an answer if you think it has answered your question $\endgroup$ – MaxU Apr 22 '18 at 18:31
  • $\begingroup$ @Yousuf, the best option would be to prepare a small reproducible data set and post it somewhere (GitHub, any file-exchange web site) and ask a new question (including your code). But the most important thing is a small reproducible data set and your desired data set... $\endgroup$ – MaxU Apr 22 '18 at 18:32
  • $\begingroup$ Doing the same thing but I wanted to try something different. I don't want unknown features to be ignored. I want them to participate in prediction. Any idea from where I can start. $\endgroup$ – Hammad Hassan Nov 1 '18 at 8:14
1
$\begingroup$

I think you would need to delete exactly those features (columns) that are not known to your model.

Imagine that you trained your model on the following features:

  • human weight
  • human height
  • sex
  • age

and now you want to feed it a test data set containing the following features:

  • human weight
  • human height
  • sex
  • age
  • eye color
  • name
  • zip code

so if if you randomly reduce it to the correct shape and get:

  • eye color (should be: human weight)
  • human height
  • zip code (should be: sex)
  • name (should be: age)

what kind of accuracy would you expect in this case?

$\endgroup$
  • $\begingroup$ Thanks for the answer. That is true. So do i have to manually delete those from my file? Or is there a function ? $\endgroup$ – Yousuf Apr 21 '18 at 18:15
  • $\begingroup$ @Yousuf, well it depends on how do you process your data... If you know indices of needed features, then it's fairly easy to implement in Numpy or Pandas ... $\endgroup$ – MaxU Apr 21 '18 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.