high precision and recall but low cross validated accuracy

Question

I'm using a random forest classifier (scikit learn). When I do a 5-fold cross validation, the average accuracy score is around 70%. However, when I look at precision_score and recall_score, they are both 1. Is that possible? If it's wrong, what could be the problem? Here is the code:

clf = RandomForestClassifier(n_estimators=100) 
clf.fit(X, Y.values.ravel())
scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")
Y_pred = clf.predict(X)
precision = precision_score(Y, Y_pred)
recall = recall_score(Y, Y_pred)

Answer 1

First of all, you should use cross_val_predict to get you predictions vector, so that you followed approximately the same validation scheme to get them :

Y_pred = cross_val_predict(clf, Y, cv=5)

Then, it is totally possible to have 1 for recall or precision given that these are the scores for the considered positive class for scikit. In fact scikit assigns by default one of your class as the 'positive' one and to metrics inherent to classes are computed according to it.

However, as you said you can't have 1 for both and 70% accuracy.

The fact is that your clf.predict gave you 100% accuracy while during cross validation it was not the case because in your second scheme:

clf.fit(X, Y.values.ravel()) and Y_pred = clf.predict(X)

you train and predict on the same data ! So you obtain 100% accuracy which is common, while during cross validation :

scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")

you trained and tested on different folds which logically will lead to worse results (but closer to real condition experiment).

Besides, try to use cross_val_predict and then precision_recall_fscore_support(Y, Y_pred) in order to have a detailed report of metrics for each class.