# very large difference between cross_val and (multiple) r2 model evaluation

I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and run an r2 score (accuracy) on the predicted train & test set:

X_train, X_test, y_train, y_test = train_test_split(
df2, y, test_size=0.33, random_state=np.random.randint(0,100))


which gives (0.9325975114210778, 0.8791805732349958) Repeatedly over multiple runs, I believe this is a reasonable score?

When I run a cross validation on the same model (same parameters) I do get a much lower score.

gradModel = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)
from sklearn.model_selection import cross_val_score
scores = cross_val_score(gradModel, df2, y, cv=15)
print ("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.45 +/- 0.21


Which I believe is a more realistic score, but I can't explain the large differences. Also because the original r2 score is more or less equal over multiple reruns

• What's your new score @dr jerry? Also r2 and accuracy aren't the same metric right? – Aditya Sep 6 '18 at 0:55
• naively assumed they would have same metric. I'll reinvestigate. Thanks! – dr jerry Sep 6 '18 at 7:09
• @Aditya crossvalidation, by default uses the r2 metric (lacking in the documentation), the same metric which is used for GradientBoostingRegressor.score. So (by coincidence) my primary assumption was correct: the cross_val, r2_score and gradModel.score are expected to give the same values approx. After investigating with co-workers we believe the difference between cross_validation and r2 is that cross_val uses a stratified split. – dr jerry Sep 13 '18 at 12:03
• Thanks Jerry for the Update! I will need to re-read a lot of stuff! – Aditya Sep 13 '18 at 12:38