1
$\begingroup$

I'm using a random forest classifier (scikit learn). When I do a 5-fold cross validation, the average accuracy score is around 70%. However, when I look at precision_score and recall_score, they are both 1. Is that possible? If it's wrong, what could be the problem? Here is the code:

clf = RandomForestClassifier(n_estimators=100) 
clf.fit(X, Y.values.ravel())
scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")
Y_pred = clf.predict(X)
precision = precision_score(Y, Y_pred)
recall = recall_score(Y, Y_pred)
$\endgroup$
0
$\begingroup$

First of all, you should use cross_val_predict to get you predictions vector, so that you followed approximately the same validation scheme to get them :

Y_pred = cross_val_predict(clf, Y, cv=5)

Then, it is totally possible to have 1 for recall or precision given that these are the scores for the considered positive class for scikit. In fact scikit assigns by default one of your class as the 'positive' one and to metrics inherent to classes are computed according to it.

However, as you said you can't have 1 for both and 70% accuracy.

The fact is that your clf.predict gave you 100% accuracy while during cross validation it was not the case because in your second scheme:

clf.fit(X, Y.values.ravel()) and Y_pred = clf.predict(X)

you train and predict on the same data ! So you obtain 100% accuracy which is common, while during cross validation :

scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")

you trained and tested on different folds which logically will lead to worse results (but closer to real condition experiment).

Besides, try to use cross_val_predict and then precision_recall_fscore_support(Y, Y_pred) in order to have a detailed report of metrics for each class.

$\endgroup$
  • $\begingroup$ OK, thanks! By the way, how would one get the predicted probabilities using the same scheme? cross_val_predict gives the class prediction through some cross validation. Is there a corresponding way to get the class probabilities? $\endgroup$ – physics_2015 Dec 4 '17 at 5:52
  • $\begingroup$ Yes ! You can normally use Y_pred = cross_val_predict(clf, Y, cv=5, method='predict_proba') $\endgroup$ – Elliot Dec 4 '17 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.