# ValueError trying to use a pickled scikit-learn model

I am new to data science and trying to learn something. I was able to complete the prediction with 98% accuracy and i saved it as pickle model. Now while trying to predict using this model I am getting the below error.

trainFile=os.path.join('D:\PYPrograms','Data','POS','collected.csv')
#load the data
train  = pd.read_csv(trainFile)
dataTemp=train
nullInTrain=train.shape[0] - train.dropna().shape[0]
print("Null values in Train data "+str(nullInTrain))
dataTemp.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
dataTemp.loc[:,"title"] = dataTemp.title.apply(lambda x : " ".join(re.findall('[\w]+',x)))
df1 = dataTemp.dropna()
cv1 = CountVectorizer()
df_x = df1["tickettype"]+" "+df1["title"]
df_y = df1["type"]
X_train, X_test, y_train, y_test = train_test_split(df_x, df_y, test_size=0.2, random_state=0)
x_traincv = cv1.fit_transform(X_train)
x_testcv = cv1.transform(X_test)
clf = RandomForestClassifier(n_estimators = 1000, max_depth = 6)
clf.fit(x_traincv,y_train)
pred=clf.predict(x_testcv)
pred
#make prediction and check model's accuracy
predictions_test = clf.predict(x_testcv)
acc =  accuracy_score(np.array(y_test),predictions_test)
print ('The accuracy of Random Forest is {}'.format(acc))
import pickle
modelFile=os.path.join('D:\PYPrograms','Data','model2')
with open(modelFile, 'wb') as picklefile:
pickle.dump(clf,picklefile)

with open(modelFile, 'rb') as training_model:
model = pickle.load(training_model)

cv2 = CountVectorizer()
File=os.path.join('D:\PYPrograms','Data','POS','Report_one_wk08.csv')
data = pd.read_csv(File)
data.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
test = cv2.fit_transform(data['title'])
model.predict(test)


# Error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-205-c0ac8462bce6> in <module>
----> 1 model.predict(test)

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
543             The predicted classes.
544         """
--> 545         proba = self.predict_proba(X)
546
547         if self.n_outputs_ == 1:

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self, X)
586         check_is_fitted(self, 'estimators_')
587         # Check data
--> 588         X = self._validate_X_predict(X)
589
590         # Assign chunk of trees to jobs

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in _validate_X_predict(self, X)
357                                  "call fit before exploiting the model.")
358
--> 359         return self.estimators_[0]._validate_X_predict(X, check_input=True)
360
361     @property

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self, X, check_input)
400                              "match the input. Model n_features is %s and "
401                              "input n_features is %s "
--> 402                              % (self.n_features_, n_features))
403
404         return X

ValueError: Number of features of the model must match the input. Model n_features is 6639 and input n_features is 3


Data available at https://drive.google.com/open?id=1xaKKSXzpr7THezqU_8jycfvAueg0nnCQ

• What is the error? – Erwan Jan 26 at 0:04
• It looks like you're not formatting the test data in the same way as the training data. You must apply the same pre-processing steps to the test data as you did on the training data, in particular the features must be exactly the same. – Erwan Jan 26 at 14:41
• Thanks for your Response @Erwan. I did try the same. If I use the same object for the countVectorize I am getting the result. When I try the new object I am getting NotFittedError: CountVectorizer - Vocabulary wasn't fitted. I am not sure with which element i need to fit. – Ricky Jan 27 at 5:10
• I'm not familiar with with scikit but apparently you're supposed to use the same CountVectorizer object for both the training and test set (stackoverflow.com/questions/44193154/…). I assume this is to ensure that the test instances are represented using the same vocabulary. – Erwan Jan 27 at 18:07

## 1 Answer

You have to use the same CountVectorizer instance on all data and have a method to handle out of training sample tokens.