Given is a dataset that I need to use for a classification and I want to compare the performance of different classification models. Let's assume, I want to look at logistic regression (with different cut-off-points) and KNN. Is there anything problematic if I proceed as follows:
- Split data in training and validation data (and a test set for the performance evaluation of the winning model).
- Train a logistic regression model and a KNN classification model on the training set. I consider for each cut-off point t between 0 and 1 the logistic regression model as a classification model - so the regression model leads to many classification models.
- I now compare for a certain range of t (lets say 0.01 to 0.99) the classification performance of all my classification models (logistic regression for those t and KNN) on the validation data. The one with the best performance (based on a certain metric) I'll choose.
I was discussing this with somebody else who argued that t needs to be considered as hyperparameter and this parameter needs to be tuned separately. If this is true - why? And what's wrong with my arguments above?