0
$\begingroup$

I know that we should use the validation set to perform hyperparameter tuning and that test dataset is not anymore really the test if it is used for hyperparameter tuning. But is this a problem if i want to compare the performance of 2 algorithms (e.g., Random Forest and XGBoost) across 10 different datasets, where each time I am using the test data for tuning. I believe that if they are trained and tested under the same conditions, the final performance analysis should be actually true representation which algorithm is better performing on these datasets. Or am i mistaken?

$\endgroup$
2
  • $\begingroup$ It's not totally clear to me what your proposed procedure is. Could you try to clarify a bit? $\endgroup$
    – Ben Reiniger
    Commented Oct 4, 2023 at 22:28
  • $\begingroup$ I have two classifiers, A and B, and I want to determine which performs better across 10 datasets. To avoid overfitting I do hyperparameter tuning of A and B -> where A and B are trained with the training data and but performance is evaluated on test data (common approach is to use validation set in model tuning...). After the best set of hyperparameters is found, I test it on test data. This is done for model A and B for each dataset and I compare final results. If A is better than B in this approach, would it be also better if I would use validation set instead of test set in tuning phase? $\endgroup$
    – John B
    Commented Oct 7, 2023 at 14:14

1 Answer 1

2
$\begingroup$

Seems like there is something flawed in the procedure here. If you use the test data set for tuning, then what do you use for testing performance?

In general, the models should not get any information from the test set. If models are exposed to the test set you will generally tend to conclude the more flexible model has better performance when it may tend to overfit the training data and underperform simpler models if the test set is isolated during the model fitting procedure.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer. In my case I have Random Forest and XGBoost. If e.g., the Random Forest performs better using this approach, does it mean that it would perform also better if I use validation set for tuning and leave the test set only for the testing phase? The purpose is only to compare these two classifiers and since they undergo the same training and testing conditions, why the results are not relevant? Thanks! $\endgroup$
    – John B
    Commented Oct 7, 2023 at 14:20
  • $\begingroup$ The only way to know for sure is to run the data pipeline and look at the results. Can you describe your data pipeline more. It sounds like you only have a training set and a validation set? Are you doing any preprocessing steps? If so please describe that process. $\endgroup$
    – noNameTed
    Commented Oct 8, 2023 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.