# ML: Classification Model Comparison

Given is a dataset that I need to use for a classification and I want to compare the performance of different classification models. Let's assume, I want to look at logistic regression (with different cut-off-points) and KNN. Is there anything problematic if I proceed as follows:

1. Split data in training and validation data (and a test set for the performance evaluation of the winning model).
2. Train a logistic regression model and a KNN classification model on the training set. I consider for each cut-off point t between 0 and 1 the logistic regression model as a classification model - so the regression model leads to many classification models.
3. I now compare for a certain range of t (lets say 0.01 to 0.99) the classification performance of all my classification models (logistic regression for those t and KNN) on the validation data. The one with the best performance (based on a certain metric) I'll choose.

I was discussing this with somebody else who argued that t needs to be considered as hyperparameter and this parameter needs to be tuned separately. If this is true - why? And what's wrong with my arguments above?

• t is the probability threshold you're using to separate classes? For instance, if t=0.3 every probability below 0.3 is classified as 0 and every instance above 0.3 is classified as 1? Jun 4 '20 at 16:09
• Yes. Logistic Regression estimates the probabilities for the two classes 0 and 1. I use the cut-off point t to decide to which class the input belongs. For t = 0.3 it's like you mentioned. Jun 4 '20 at 16:15

I would follow the following procedure:

1. Split data into training and test datasets (and also validation set if you do not want to do k-fold cross-validation)
2. Train different models using k-fold cross-validation to also find the best hyperparameters. One of the hyperparameters could be the discrimination threshold (cut-off point) that you talked about it.
3. Use the models for the prediction of the test dataset to evaluate the performance of the models based on the unseen dataset. Now, you can choose the best model.

The general model selection is a little different and you need to use a statistical test as explained in this post

W.R.T the cut-off point, it should be noted that any parameters that it is not estimated using the training dataset it is considered as the hyperparameters.

You can compare the performance of all of your models considering different cut-off points. But that is not an efficient way. It would be better to compare the performance of the models in their best performance. It would be easier to find out in which case (i.e., with which hyperparameters and cut-off point) the model has the best performance and compares it with the other models in their best performance.

• Thanks. Actually I do not understand yet, why it is better to first choose the hyperparameter and compare only then the logistic classification model with the other models like KNN. What's the reason for this suggestion? My point is: For each cut-off t, the logistic regression is a classification model, so I can compare the performance of all my models on the validation set and choose the best one. Jun 4 '20 at 18:32
• I added a few lines. Hopefully, they are convincing! Jun 4 '20 at 18:43
• Thanks nimar. Let's assume efficiency is not relevant. Is there something methodological wrong about my approach? Does my approach maybe leed to a biased choice? Jun 4 '20 at 18:55
• Are you going to compare the performance of model A (e.g., logistic regression) with model B (e.g., KNN) given different cut-off points? For example, one time with cut-off point of 0.5, one time with cut-off point of 0.55, etc. Jun 4 '20 at 19:33
• Exactly. Let's say, with logistic regression I get for each cut-off-point t a classification model m(t). Then I compare all the models m(t) and KNN on the validation set. (Actually, I would choose not all the infinitely many t but only a range like 0.01 to 0.99. So I would compare in total 100 models: m(0.01), m(0.02),...,m(0.99), KNN). Jun 4 '20 at 20:12

In principle you can use your approach.

However, you should not optimize on your test set (step 3). Instead you should select the best t using your validation set. Then you compare it against KNN, also on the validation set. Finally, the best model should be evaluated on the test set.

• Comments are not for extended discussion; this conversation has been moved to chat. Jun 4 '20 at 20:31
• Maybe KNN as lazy learner was not the best example here...let's say, we take a decion tree, a nn or an svm instead. Jun 4 '20 at 20:31
• There is one subtle issue with KNN. You cannot train it on set A and perform inference on a set B using the training of set A. Jun 4 '20 at 20:32

DON'T USE ACCURACY! USE PROPER SCORING RULES!

What you propose is related to the area under the receiver operator curve, ROCAUC. ROCs plot the sensitivity and specificity (really 1-specificity) at all possible threshold cutoffs.

It sounds like you would pick the model that has the highest accuracy value, regardless of that threshold. If the best accuracy comes from logistic regression with a threshold of $$0.6$$, go with that model. If the best accuracy comes from KNN with a threshold of $$0.07$$, go with that model.

That sounds great, right, picking the most accurate model?

THIS IS INCORRECT, tempting as it sounds. Here are a few blog posts on this topic by a professor at Vanderbilt University and an active member on Cross Validated (the statistics Stack).

https://www.fharrell.com/post/class-damage/

https://www.fharrell.com/post/classification/

(Frank Harrell even has a post about how ROCAUC is flawed for model comparisons.)

Accuracy is a flawed performance metric. Any performance metric based on a threshold has considerable flaws. Please refer to this excellent post on the topic.

Shamelessly, I will link a question I posted on a similar topic that was answered by the same person with the same gist. Here is yet another post of his on this topic.

(I plan to accept that answer but don't want to yet so others might post their thoughts.)

An easy proper scoring rule to get you started is Brier score, basically square loss. Take the probability of being in class $$1$$, subtract the true class ($$0$$ or $$1$$), square that value, and add up those values for each prediction.

$$Brier(y,\hat{p}) = \sum_{i=1}^N \big(y_i-\hat{p}_i \big)^2$$

$$y_i$$ is the true class, $$0$$ or $$1$$, and $$\hat{p}_i$$ is the predicted probability (which will most likely be the predicted probability of being in class $$1$$). You can adjust Brier score if your software gives you the probability of being class $$0$$.

• Actually, I think that the choice of the actual error metric is irrelevant to my question: In principle, you can consider the conofusion matrix and calculate the total costs of FP and FN. Then you compare the models based on this cost function. Jun 4 '20 at 20:38
• This is exactly the type of cost function argued against by Harrell and Kolassa. In order to get a confusion matrix, you have to pick a threshold. The error metric is exactly what your question asks about.
– Dave
Jun 4 '20 at 20:41
• Thanks for the references. I will take a look at them. Jun 4 '20 at 20:44

I was discussing this with somebody else who argued that t needs to be considered as hyperparameter and this parameter needs to be tuned separately.

In your exercise, you are actually doing the same thing. Getting the best t. So, I don't think you need anything extra.

What I see missing in your steps -