4
$\begingroup$

I have been working on binary classification problem using algorithms such as Random Forest, neural networks, Boosting methods and logistic regression.

However, during my model building process, I tweaked my model based on the performance in test set (X_test). Meaning, I do the below

step-1) I apply .fit() on train data, assess the performance (identify best parameters through grdisearchcv)

step-2) Later, I apply .predict() on test_data

When performance was not good on test_data, I did the below

a) Changed the algorithm (or hyperparameters,cv folds, scoring etc) and repeated step 1) and step 2)

While I found out by reading online that this is not a good approach as I am exposing the model to test data (multiple times) and model may overfit for my test_data (and not perform well in future for new data from real world).

So, now I want to erase my model's memory/make it unsee whatever it has already seen.

How can I reset ML model memory? Does resetting my jupyter notebook, laptop etc would make it forget everything?

$\endgroup$

1 Answer 1

2
$\begingroup$

Your mistake is that you make adjustments based on the test data performance and then retest on the same test data when you think you’ve made an improvement.

In “regular” machine learning, say a linear regression, you fiddle with the regression parameters until you find a minimal loss value. That’s essentially what you’re doing here. You fiddle with the model hyperparameters on the training data and the test them out on the test data. This risks overfitting the hyperparameters to the test data in the same way that parameters fit to the in-sample data.

In other words, you risk tuning your hyperparameters to fit the test data, rather than giving good ability to generalize.

$\endgroup$
3
  • $\begingroup$ Yes, now that I have done the mistake, how can I make the model unsee (what it saw)? Is it even possible? $\endgroup$
    – The Great
    Feb 25, 2022 at 12:05
  • $\begingroup$ I read somehwere that test data should be treated like as if it is vault. So, now I want to go back to 1st stage where model doesn't know how my test data looks like $\endgroup$
    – The Great
    Feb 25, 2022 at 12:06
  • $\begingroup$ The model doesn’t “see” anything. All the model does is aim for the lowest loss value it can achieve. However, since you have seen a performance value that you dislike, you now are tuning your hyperparameters to fit the test data, rather than going for generalizability (or at least you risk this). This is an example of what Fisher meant when he said that a statistician called in at the end of an experiment could do no more than an autopsy to reveal the experiment’s cause of death (though your situation might not be quite that severe yet). $\endgroup$
    – Dave
    Feb 25, 2022 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.