1
$\begingroup$

I've been trying to complete this regression task on Kaggle. As usual they gave a train.csv(with response variable) and a test.csv (without response variable) file for us to train the model and compute our predictions, respectively.

I further split the train.csv file into a train_set and test_set. I use this subsequent train_set to train a list of models which I will then shortlist to one model only based on 10-fold cross validation scores (RMSLE) and after hyperparameter tuning. Now I have one best model, which is Random Forest (with best hyperparameters) with an average RMSLE score of 0.55. At this point I have NOT touched the test_set.

Consequently, when I train the same exact model on train_set data, but evaluate its result on test_set (in order to avoid overfitting the hyperparameters I have tuned), it yields an RMSLE score of 0.54. This is when I get suspicious, because my score on test_set are slightly better than the average score of the train_set (test_set results are supposed to be slightly worse, since the model hasn't seen the test_set data, right?).

Finally, I proceed to submit my results using the same model but with the test.csv file (without response variable). But then Kaggle gave me an RMSLE score of 0.77, which is considerably worse than my cross validation scores and my test_set scores!

I am very frustrated and confused as to why this would happen, since I believe I've taken every measure to anticipate overfitting my model. Please give a detailed but simple explanation, I'm still a beginner so I might not understand overly technical terms.

$\endgroup$
  • $\begingroup$ I hate it when folks down-vote without explanation, especially to a new member. Welcome. The Kaggle folks don't give you the full set. They rank you on a hidden set. Sometimes that hidden set is very different than the train/valid/test set. Cheaters have found ways to exfiltrate parts of the hidden set, which is illegal, and then optimize for that. This is another part of the disconnect. I use train/test to decide structure and parameters, and to estimate final performance, but then I train the production model on all the information that I have. $\endgroup$ – EngrStudent Aug 14 at 19:07
  • $\begingroup$ Does that mean the process / method that I used to train and evaluate my model is correct? I just knew that this Kaggle’s ‘hidden set’ could be very different from the given dataset. $\endgroup$ – imavv Aug 14 at 19:22
  • $\begingroup$ The "hidden set" only exists in Kernel Only competitions. $\endgroup$ – Pedro Henrique Monforte Aug 15 at 1:11
  • 1
    $\begingroup$ @imavv - there is no silver bullet. You can talk to stats guys and even if you do it correct, there are an infinite number of more correct things to do. You need to grow a feel for correct enough, and that comes from pushing that rock called better and better up the big hill every day for a lot of days in a row. You can do things "right" and fail because the thing was not the tool that could crack that nut. $\endgroup$ – EngrStudent Aug 15 at 14:53
  • $\begingroup$ Thank you for the insight. This has greatly helped. $\endgroup$ – imavv Aug 16 at 1:47
2
$\begingroup$

This "train split" you named train_set and test_set are not guarantee to be clean or even balanced.

When your test set has better performance than your training set that might mean that you have data leakage (some examples in the test set are equal to the training set) or just mean your test set is slightly easier than the training set.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Personally I don't look much at absolute scoring of the two, but I look at the trends over variation of learner parameter. This tells me where the "cliff" is, and where the "peak" is. $\endgroup$ – EngrStudent Aug 15 at 14:51
0
$\begingroup$

They want to test your ability to generalise.

Test (holdout/leaderboard) set will Always have somewhat different Distribution (i.e. covariant shift) hence often you have a leaderboard shakeup. Also often People try to probe the test set to learn this Distribution and adjust the model accordingly.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.