1
$\begingroup$

Jane trains three different classifiers: Logistic Regression, Decision Tree, and Support Vector Machines on the training set. Each classifier has one hyper-parameter (regularisation parameter, depth-of-tree, etc) that needs to set, so she chooses a set of 10 reasonable values for each and performs a sweep over those values (retraining the classifier each time) and chooses the value that gives the best performance on the test set. She then reports performance for the three classifiers with their best hyper-parameter settings on the test dataset. Is there a problem with her approach? I think the answer regards some problems related to overfitting ,isn't it?

$\endgroup$

1 Answer 1

1
$\begingroup$

This is called data leakage.

She should choose hyper parameters based upon the training data. And then report performance when evaluated using the test data.

Being able to make good predictions on already seen data is uninteresting . Any relational database could accomplish that trick, it is trivial. What we care about is the ability of a model to generalize to unseen data, and to make good predictions on that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.