1
$\begingroup$

I am new to data science and trying to learn something. I was able to complete the prediction with 98% accuracy and i saved it as pickle model. Now while trying to predict using this model I am getting the below error.

trainFile=os.path.join('D:\PYPrograms','Data','POS','collected.csv')
#load the data
train  = pd.read_csv(trainFile)
dataTemp=train
nullInTrain=train.shape[0] - train.dropna().shape[0]
print("Null values in Train data "+str(nullInTrain))
dataTemp.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
dataTemp.loc[:,"title"] = dataTemp.title.apply(lambda x : " ".join(re.findall('[\w]+',x)))
df1 = dataTemp.dropna()
cv1 = CountVectorizer()
df_x = df1["tickettype"]+" "+df1["title"]
df_y = df1["type"]
X_train, X_test, y_train, y_test = train_test_split(df_x, df_y, test_size=0.2, random_state=0)
x_traincv = cv1.fit_transform(X_train)
x_testcv = cv1.transform(X_test)
clf = RandomForestClassifier(n_estimators = 1000, max_depth = 6)
clf.fit(x_traincv,y_train)
pred=clf.predict(x_testcv)
pred
#make prediction and check model's accuracy
predictions_test = clf.predict(x_testcv)
acc =  accuracy_score(np.array(y_test),predictions_test)
print ('The accuracy of Random Forest is {}'.format(acc))
import pickle
modelFile=os.path.join('D:\PYPrograms','Data','model2')
with open(modelFile, 'wb') as picklefile:
    pickle.dump(clf,picklefile)

with open(modelFile, 'rb') as training_model:
    model = pickle.load(training_model)

cv2 = CountVectorizer()
File=os.path.join('D:\PYPrograms','Data','POS','Report_one_wk08.csv')
data = pd.read_csv(File)
data.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
test = cv2.fit_transform(data['title'])
model.predict(test)

Error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-205-c0ac8462bce6> in <module>
----> 1 model.predict(test)

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
    543             The predicted classes.
    544         """
--> 545         proba = self.predict_proba(X)
    546 
    547         if self.n_outputs_ == 1:

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self, X)
    586         check_is_fitted(self, 'estimators_')
    587         # Check data
--> 588         X = self._validate_X_predict(X)
    589 
    590         # Assign chunk of trees to jobs

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in _validate_X_predict(self, X)
    357                                  "call `fit` before exploiting the model.")
    358 
--> 359         return self.estimators_[0]._validate_X_predict(X, check_input=True)
    360 
    361     @property

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self, X, check_input)
    400                              "match the input. Model n_features is %s and "
    401                              "input n_features is %s "
--> 402                              % (self.n_features_, n_features))
    403 
    404         return X

ValueError: Number of features of the model must match the input. Model n_features is 6639 and input n_features is 3 

Data available at https://drive.google.com/open?id=1xaKKSXzpr7THezqU_8jycfvAueg0nnCQ

$\endgroup$
4
  • 1
    $\begingroup$ What is the error? $\endgroup$
    – Erwan
    Jan 26 '20 at 0:04
  • 2
    $\begingroup$ It looks like you're not formatting the test data in the same way as the training data. You must apply the same pre-processing steps to the test data as you did on the training data, in particular the features must be exactly the same. $\endgroup$
    – Erwan
    Jan 26 '20 at 14:41
  • $\begingroup$ Thanks for your Response @Erwan. I did try the same. If I use the same object for the countVectorize I am getting the result. When I try the new object I am getting NotFittedError: CountVectorizer - Vocabulary wasn't fitted. I am not sure with which element i need to fit. $\endgroup$ Jan 27 '20 at 5:10
  • $\begingroup$ I'm not familiar with with scikit but apparently you're supposed to use the same CountVectorizer object for both the training and test set (stackoverflow.com/questions/44193154/…). I assume this is to ensure that the test instances are represented using the same vocabulary. $\endgroup$
    – Erwan
    Jan 27 '20 at 18:07
1
$\begingroup$

You have to use the same CountVectorizer instance on all data and have a method to handle out of training sample tokens.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.