3
$\begingroup$

Every thread on stackexchange that I've found says that you can only use the test set once and thats it. So for instance, if you used a linear regression model and got poor results on the test set, you cannot change the the model to say a random forest and evaluate this model again on the test set.

This doesnt make sense to me. Example, we all know the MNIST dataset well. Lets say I download the data and split it into train, validation and test. Also, say I used a linear regression model and clearly I will do poorly on the test set. Now, what's stopping anyone else from downloading the same MNIST data set, splitting into train, validation, test and using a different model (e.g. neural network) and reporting their test result?

According to our understanding of only being able to use the test set once, their test results are invalid because they have somehow "learnt" that a linear regression model was not good. This does not seem right to me.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

The reason for the train/validate/test distribution (which I found out in a painful way) is that you will get good results if you tweak a model to fit to a test set. It could be completely random data, but if you calculate enough features, and tweak the hyperparameters of your model, you will get a relatively high (and misleading) level of accuracy.

not tweaking to your test set is a rule of thumb, which people treat like gospel because they've been burned by not following it. In reality, the more nuanced advise is this:

  1. Train on a test set
  2. test your model on a validation set
  3. tweak your model based on the validation results
  4. once you feel confident, test on a completely external test set.

If you get bad results, you can go back to steps 1-3, but you should be careful not to use your test results from 4 to tweak your model, as that defeats the whole point of separating the validation and test set in the first place.

there are systems that automate this process, like train/validate/test splits and nested kfold.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your answer! But dont you agree that by going back to steps 1-3 that is considered as tweaking your model to your test set? In a way, you are changing your model now to get a lower test set error. And if you keep on doing this you are overfitting on your test error. Then it seems like you need another test set to check for this overfitting, and then you can kind of see that the cycle repeats itself... $\endgroup$
    – woowz
    Commented Aug 30, 2021 at 16:26
  • 1
    $\begingroup$ This is where the art comes in. When you're working in the real world, these rules of thumb often don't fit perfectly. I use test results to change how I approach fitting to the validation group. For instance, I'm overfitting to my validation set based on test results, maybe the test results have a hard time with specific class, etc. It's kind of like plagiarism; in general it's wrong, but in reality the line between adapting knowledge and plagiarizing can be murky. In my experience, it's not an issue if you use your test results to paint changes with a large brush infrequently. $\endgroup$
    – Warlax56
    Commented Aug 30, 2021 at 17:18
  • $\begingroup$ You can also have a sanity check dataset on the side, which I sometimes do. The painful reality is, getting nitty gritty with exactly how you do train/validate/test splits matters the most when you have a small amount of data, and reserving large amounts of data is infeasible. In these scenarios, reserving large amounts of data can inhibit performance. If you have enough data where this is not an issue, and your test set includes millions of records, you might not even care about "overfitting" to your test set because it's so diverse. It all has a lot to do with the data you're working with. $\endgroup$
    – Warlax56
    Commented Aug 30, 2021 at 17:21
  • $\begingroup$ At the end of the day, it's intractable, which you find a lot in computer science. Like the Two Generals Problem, sometimes things are turtles all the way down, and you have to gauge your level of confidence based on domain knowledge, experience, and best practices. $\endgroup$
    – Warlax56
    Commented Aug 30, 2021 at 17:27
  • 2
    $\begingroup$ yup I do agree with you it is an art! I think I understand now. We have to have different strategies when dealing with large or small data sets. For really large data sets, overfitting will be less of an issue, so it'll be ok to go back and tweak your model again if you get bad results. For really small data sets, we should keep overfitting in mind. If we tweak our model again, we need to just be aware that the good result you obtain might be optimistically biased and in reality you may perform much worse! $\endgroup$
    – woowz
    Commented Aug 30, 2021 at 18:30
0
$\begingroup$

You can test multiple different models. In fact, in industry, it's pretty common to start out with a simple model and then build a more advanced model. The first test of a simple model might be a K-NN. Then you might build something more advanced like a random forest.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, thats seems like whats done, because anyone can create a train, validation and test split on any data. But still, why is it that people say you can only use the test set once? This seems like we are using the test set multiple times (from different people creating different random test sets). $\endgroup$
    – woowz
    Commented Aug 30, 2021 at 3:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.