1
$\begingroup$

I have a train and a test set and no development (dev) set. I'm training a model on the train set and searching for the best hyperparameters that can eventually maximize the accuracy of the test set (pretty much a normal machine learning scenario). Here is my confusion: we usually do the hyperparameter tuning on the development set (not test set) to find the best hyperparameters, then we use those best hyperparameters to train our model and finally test it on the test set. I have two questions though when there is no dev set:

  • Is it problematic if we do hyperparameter searching on the test set? One may say, it is obviously problematic, but I say the hyperparameter searching is like every time training a model from scratch using a combination of hyperparameters and pick the best combination, and it's not like that the model is learning from previous hyperparameter searches, so is this still problematic?
  • If the first option is problematic, should I just break my train set into train+dev set and then use the dev for hyperparameter searching?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This is very nearly a duplicate of Is a test set necessary after cross validation on training set?, but I think it's worth addressing specifically this part of your question:

the hyperparameter searching is like every time training a model from scratch using a combination of hyperparameters and pick the best combination, and it's not like that the model is learning from previous hyperparameter searches

Indeed the various models don't gain any direct information from each other, either from training or the dev set. However, you are choosing the hyperparameters that perform best on the dev set, so the whole pipeline can become "overfit" to the dev set. The size of that effect is most often not too pronounced, but you certainly cannot report the score of the chosen hyperparameters as an unbiased estimate of future performance.

The answer to your second question is then "yes"; and if your training set is small, consider a repeated split cross-validation like k-fold.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for pointing to the very nearly duplicate of my post, couldn't find that post before posting my question. And, thanks for the response, I think based on your response and reviewing all responses in the other post, my conclusion is that hyperparameter searching using the test set can be considered as leakage and I should either create a dev set or go with cross-validation on my train set for the hyperparameter tuning. $\endgroup$
    – Pedram
    Jun 28, 2021 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.