You should not experience this problem if you use TfidfVectorizer properly.
Demo:
In [58]: from sklearn.feature_extraction.text import TfidfVectorizer
source text
In [59]: text = """I trained a classifier using TfidfVectorizer in Sklearn. I then pickled the model for future use.
...:
...: The new x_test that I want to make predictions on, has more features than the x_train from the model. This is the resulting error"""
let's tokenize it to a list of sentenses:
In [60]: from nltk import sent_tokenize
In [61]: vect = TfidfVectorizer()
In [62]: data = sent_tokenize(text)
yields:
In [63]: data
Out[63]:
['I trained a classifier using TfidfVectorizer in Sklearn.',
'I then pickled the model for future use.',
'The new x_test that I want to make predictions on, has more features than the x_train from the model.',
'This is the resulting error']
now we can fit and transform our data set:
In [64]: X = vect.fit_transform(data)
result:
In [65]: X
Out[65]:
<4x31 sparse matrix of type '<class 'numpy.float64'>'
with 34 stored elements in Compressed Sparse Row format>
In [66]: vect.get_feature_names()
Out[66]:
['classifier',
'error',
'features',
'for',
'from',
'future',
'has',
'in',
'is',
'make',
'model',
'more',
'new',
'on',
'pickled',
'predictions',
'resulting',
'sklearn',
'tfidfvectorizer',
'than',
'that',
'the',
'then',
'this',
'to',
'trained',
'use',
'using',
'want',
'x_test',
'x_train']
now let's feed it a data set with unknown words (features):
In [67]: new_dataset = ["let's see what happens to unknown words", "Yet another sentence."]
In [68]: X2 = vect.transform(new_dataset)
In [69]: X2
Out[69]:
<2x31 sparse matrix of type '<class 'numpy.float64'>'
with 1 stored elements in Compressed Sparse Row format>
it worked properly - all unknown features (words) have been ignored:
In [70]: pd.SparseDataFrame(X2, columns=vect.get_feature_names(), default_fill_value=0)
Out[70]:
classifier error features for from future has in is make ... the then this to trained use using want \
0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
x_test x_train
0 0.0 0.0
1 0.0 0.0
[2 rows x 31 columns]