How to compare models and which settings to keep constant? [closed]

Question

I already posted this in another forum but no response. So, posting it here.

Currently, in clinical practice, clinicians use a score (as a single feature) to predict the mortality of a patient. Now in my project based on clinician inputs, we have created two new features to see whether we can enhance the prediction accuracy. Since our objective is to predict yes or no, I tried logistic regression and got the below results

Now my question is as below

a) Since our dataset of 3.2k rows is imbalanced (87:13), I tried to optimize the decision threshold for classification. But the optimized decision threshold value changes based on the feature set. So, I guess that's expected and my comparison is still valid. For example, I am comparing model A with only one feature (existing_feature) with model B with two features (existing_feature and new_feat_1) and model C with two features (existing_feature, new_feat_1, and new_feat_2). Is my comparison valid? I can't have the same threshold because when I have a different no of features, the optimal threshold will definitely change. I can't just choose 0.5 as the default because my dataset is imbalanced (87:13). So any advice, please? Should I change or not change the optimal threshold for different models?

b) Since my dataset is imbalanced, am focusing on f1-score as an evaluation metric. We can see that there is some improvement in the f1-score due to the addition of 2 new features. How can I know that these 2 new features are indeed useful (and add value to prediction) and not just by some random chance? Any suggestion on how to assess the usefulness of this feature? Please note that am using the random_state variable in the scikit-learn log regression function. So, everything else is controlled. It's just that am adding new features and changing the optimal threshold for it. I don't change anything else.

c) In a hospital setting, False negatives are costly. Meaning if we miss predicting a person who will die at a high risk of dying, it is costly. So, I guess we have to look at precision for it. However, you can see that my recall is dropping heavily. So, how should I decide whether this model and new features are helpful or not

d) For the purpose of interpretability, I used only the logistic regression model. Do you advise me to run other models like SVM, RF, and Xgboost, etc? Since my dataset is imbalanced and non-linear separation, do you think it's good to try other models as well. or based on your experience, looking at the results, do you think there cannot be any further improvement to this?

e) I am using the class_weight=balanced parameter to run my log reg model as my dataset is balanced. Do you advise me to oversample the minority class? In real-time, people who are dead are always less when compared to people who are alive (for the problem that I am studying). Meaning, the positive class will always be the minority class. So should I oversample it or just use the class_weight=balanced parameter and carry on with my tasks?

f) I can use gridsearchCV and find the best parameter to be used for model A. Can the same parameters be best for model B? Do I have to run gridsearchCV for each of the models A, B, and C to identify the best parameters. If I am going to use different best parameters for each model, then I cannot compare them for performance. Am I right? Because I am changing the hyperparameters which violate controlled settings criteria for model comparison and evaluation. How should I do it?

Answer 1

General remark: your two new models give very different results in terms of precision and recall, I find this a bit surprising. I would probably try different learning methods (e.g. decision trees, SVM) in order to investigate if this is really due to the features or not.

a) Absolutely, the threshold should be specific to the model and features, it would be suboptimal to keep the same value.

b) Typically one would check whether there is a statistically significant difference in the performance between the models. I'm not sure which significance test is appropriate here.

c) Then you should evaluate your models with $F_{\beta}$-score instead of just F1, with $\beta$ higher than 1 in order to favour recall. I'd say at least 2, it depends how much more costly is a FN than a FP error. Note that the threshold should also be selected based on this measure.

d) I would certainly try different methods, because it's always possible that another method would perform better. I always recommend decision trees because they are robust and interpretable.

e) I'm not sure but there might be a confusion here: class_weight=balanced means that you give a higher importance to the minority class than what it really represents in the data, in other words the learning algorithm will work as if the two classes have the same number of instances. So as far as I know there would be no point in resampling additionally to using the weights. However you could provide manual weights, for instance 0.1,0.9 if you want to favour detecting the minority class even more (that would increase recall). By the way I would suggest to also try without using weights at all: it's unlikely to be good in terms of recall but that it could be useful just to know the precision/recall in this case.

f) For tuning hyper-parameters the important point is to use a validation set different from the final test set. For every "method" (set of features) independently:

1. Use grid search to determine the best parameters for the method, evaluating on the validation set only.
2. Pick the best parameters. At this stage if you want you can train the final model using both training set+validation set (but only with the selected parameters).

Then you can apply the 3 final models on the unseen test set and compare their performance. Essentially the parameter tuning is part of the training process (it's a kind of "meta-training"), so there's no data leakage as long as you don't use the test set for determining the best parameters.