I already posted this in another forum but no response. So, posting it here.
Currently, in clinical practice, clinicians use a score (as a single feature) to predict the mortality of a patient. Now in my project based on clinician inputs, we have created two new features to see whether we can enhance the prediction accuracy. Since our objective is to predict yes or no, I tried logistic regression and got the below results
Now my question is as below
a) Since our dataset of 3.2k rows is imbalanced (87:13), I tried to optimize the decision threshold for classification. But the optimized decision threshold value changes based on the feature set. So, I guess that's expected and my comparison is still valid. For example, I am comparing model A with only one feature (existing_feature) with model B with two features (existing_feature and new_feat_1) and model C with two features (existing_feature, new_feat_1, and new_feat_2). Is my comparison valid? I can't have the same threshold because when I have a different no of features, the optimal threshold will definitely change. I can't just choose 0.5 as the default because my dataset is imbalanced (87:13). So any advice, please? Should I change or not change the optimal threshold for different models?
b) Since my dataset is imbalanced, am focusing on f1-score as an evaluation metric. We can see that there is some improvement in the f1-score due to the addition of 2 new features. How can I know that these 2 new features are indeed useful (and add value to prediction) and not just by some random chance? Any suggestion on how to assess the usefulness of this feature? Please note that am using the random_state variable in the scikit-learn log regression function. So, everything else is controlled. It's just that am adding new features and changing the optimal threshold for it. I don't change anything else.
c) In a hospital setting, False negatives are costly. Meaning if we miss predicting a person who will die at a high risk of dying, it is costly. So, I guess we have to look at precision for it. However, you can see that my recall is dropping heavily. So, how should I decide whether this model and new features are helpful or not
d) For the purpose of interpretability, I used only the logistic regression model. Do you advise me to run other models like SVM, RF, and Xgboost, etc? Since my dataset is imbalanced and non-linear separation, do you think it's good to try other models as well. or based on your experience, looking at the results, do you think there cannot be any further improvement to this?
e) I am using the class_weight=balanced parameter to run my log reg model as my dataset is balanced. Do you advise me to oversample the minority class? In real-time, people who are dead are always less when compared to people who are alive (for the problem that I am studying). Meaning, the positive class will always be the minority class. So should I oversample it or just use the class_weight=balanced parameter and carry on with my tasks?
f) I can use gridsearchCV and find the best parameter to be used for model A. Can the same parameters be best for model B? Do I have to run gridsearchCV for each of the models A, B, and C to identify the best parameters. If I am going to use different best parameters for each model, then I cannot compare them for performance. Am I right? Because I am changing the hyperparameters which violate controlled settings criteria for model comparison and evaluation. How should I do it?