How to choose between different models with similar results? RF, GLM and XGBoost

Question

I am a medical doctor trying to make prediction models based on a database of approximately 1500 patients with 60+ parameters each.

I am dealing with a classification problem (mortality at 1, 3, 6 and 12 months) and have made stratified splits (70 training/ 30 testing) and have done feature selection with the Boruta algorithm before training Random forest, GLM and eXtreme Gradient Boosting models for each timepoints. Hyperparameter tuning was done with a gridsearch and 10-fold CV on the training part of the data for the XGB and RF models.

The AUC for all models is about 0.80 (RF model slightly better), Brier scores between 0.09-0.17 for the RF and between 0.13-0.23 for the other two.

So based on the Brier scores it seems that the RF models has a slight advantage but I am wondering:

-Should I do more performance measurements? Which ones and why?

-How to interpret my results? My understanding is that there seems to be a linear association between the predictors as the GLM model performs well, but still the RF has a slight advantage in performance and accuracy but has the disadvantage of being a more "complicated" model.

I plan to do external validation with a different dataset but as of now I would be very interested in understanding if other measurements could shed some light on advantages of the different models and also I am sure I am missing something as I am new to the field and would be very interested to hear any advice/ opinions.

Thanks!

Answer 1

Should I do more performance measurements? Which ones and why?

Generally speaking, the choice of performance metric should be informed by your goals and the characteristics of your dataset. If you care about maximizing raw accuracy, then brier score is a good metric.

However there are many situations (especially in the medical field) where maximizing raw accuracy is not an appropriate goal. A classic example is cancer screening. False positives - i.e. you tell the patient they have cancer but it is later discovered they do not - are much less harmful than false negatives - i.e. the cancer goes undiscovered and is allowed more time to spread.

Another pitfall with raw accuracy is inbalanced datasets (also quite common in the medical field). A classic example here is credit card fraud detection. Because the vast majority of transactions are not fraudulent, a model can achieve over 99% accuracy just by classifying every transaction as non-fraudulent.

All of this to say, you should consider the details of your problem and choose a performance measurement that actually measures what you are concerned about. Think about the impacts of correct vs incorrect predictions and account for biases in your dataset. Some other suggestions might be precision, recall, and f-score

How to interpret my results? My understanding is that there seems to be a linear association between the predictors as the GLM model performs well, but still the RF has a slight advantage in performance and accuracy but has the disadvantage of being a more "complicated" model.

Sounds about right! Since this is a classification problem, it might be a little more precise to say that the classes are nearly linearly separable. Saying "linear association" makes me think it's a regression problem. But that's a tiny nitpick

One confusing wrinkle in your results is the difference between RF and XGB. Both are tree ensemble models, and gradient boosting often outperforms RF. It might be interesting to find out why these two are exhibiting such different performance.