I have been working on binary classification problem using algorithms such as Random Forest, neural networks, Boosting methods and logistic regression.
However, during my model building process, I tweaked my model based on the performance in test set (
X_test). Meaning, I do the below
step-1) I apply .fit() on train data, assess the performance (identify best parameters through grdisearchcv)
step-2) Later, I apply .predict() on test_data
When performance was not good on test_data, I did the below
a) Changed the algorithm (or hyperparameters,cv folds, scoring etc) and repeated step 1) and step 2)
While I found out by reading online that this is not a good approach as I am exposing the model to test data (multiple times) and model may overfit for my test_data (and not perform well in future for new data from real world).
So, now I want to erase my model's memory/make it unsee whatever it has already seen.
How can I reset ML model memory? Does resetting my jupyter notebook, laptop etc would make it forget everything?