I did submit my first kaggle kernel, on the avocado dataset kernel link, I treated it like I should predict the avocado price so I splitted the dataset in a train & test set, fitted the model and run an r2 score (accuracy) on the predicted train & test set:

X_train, X_test, y_train, y_test = train_test_split(
    df2, y, test_size=0.33, random_state=np.random.randint(0,100))
gradModel = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)
(r2_score(y_train, gradModel.predict(X_train)),r2_score(y_test, y_pred))

which gives (0.9325975114210778, 0.8791805732349958) Repeatedly over multiple runs, I believe this is a reasonable score?

When I run a cross validation on the same model (same parameters) I do get a much lower score.

gradModel = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)
from sklearn.model_selection import cross_val_score
scores = cross_val_score(gradModel, df2, y, cv=15)
print ("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.45 +/- 0.21

Which I believe is a more realistic score, but I can't explain the large differences. Also because the original r2 score is more or less equal over multiple reruns

  • $\begingroup$ What's your new score @dr jerry? Also r2 and accuracy aren't the same metric right? $\endgroup$ – Aditya Sep 6 '18 at 0:55
  • $\begingroup$ naively assumed they would have same metric. I'll reinvestigate. Thanks! $\endgroup$ – dr jerry Sep 6 '18 at 7:09
  • 1
    $\begingroup$ @Aditya crossvalidation, by default uses the r2 metric (lacking in the documentation), the same metric which is used for GradientBoostingRegressor.score. So (by coincidence) my primary assumption was correct: the cross_val, r2_score and gradModel.score are expected to give the same values approx. After investigating with co-workers we believe the difference between cross_validation and r2 is that cross_val uses a stratified split. $\endgroup$ – dr jerry Sep 13 '18 at 12:03
  • $\begingroup$ Thanks Jerry for the Update! I will need to re-read a lot of stuff! $\endgroup$ – Aditya Sep 13 '18 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.