Number of features of the model must match the input. Model n_features is 740 and input n_features is 400

Question

i am getting this error predicting from random classifier, could anybody point me to where i am going wrong in this?

(background information: yes, i am trying to do sentence classification with 2 labels)

#Initializing BoW
cv = CountVectorizer()

#Test-Train Split
X_train,X_test,y_train,y_test = train_test_split(experiment_df['Sentence'],experiment_df['Label'])

#Transform
train = cv.fit_transform(X_train)
test = cv.fit_transform(X_test)


#Train Classifier
clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(train,y_train)

#Pred
y_pred = clf.predict(test)

Error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-31-a6f8e9da0bb0> in <module>()
      1 clf = RandomForestClassifier(max_depth=2, random_state=0)
      2 clf.fit(train,y_train)
----> 3 y_pred = clf.predict(val)

3 frames
/usr/local/lib/python3.7/dist-packages/sklearn/tree/_classes.py in _validate_X_predict(s

elf, X, check_input)
    389                              "match the input. Model n_features is %s and "
    390                              "input n_features is %s "
--> 391                              % (self.n_features_, n_features))
    392 
    393         return X

ValueError: Number of features of the model must match the input. Model n_features is 740 and input n_features is 400 
```

Answer 1

You are currently using the fit_transform method on both your training dataset as well as your test set. This is incorrect since you should not fit the model on your test set as (depending on the model used) this would be overfitting, and it can give issues with dataset shapes when creating new columns based on the values in the data (count vectorizer, creating dummy columns etc.). The correct way would be to fit and transform the train data and then only transform the test dataset:

#Initializing BoW
cv = CountVectorizer()

#Test-Train Split
X_train,X_test,y_train,y_test = train_test_split(experiment_df['Sentence'],experiment_df['Label'])

#Transform
train = cv.fit_transform(X_train)
test = cv.transform(X_test)


#Train Classifier
clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(train,y_train)

#Pred
y_pred = clf.predict(test)