I am a medical doctor trying to make prediction models based on a database of approximately 1500 patients with 60+ parameters each.
I am dealing with a classification problem (mortality at 1, 3, 6 and 12 months) and have made stratified splits (70 training/ 30 testing) and have done feature selection with the Boruta algorithm before training Random forest, GLM and eXtreme Gradient Boosting models for each timepoints. Hyperparameter tuning was done with a gridsearch and 10-fold CV on the training part of the data for the XGB and RF models.
The AUC for all models is about 0.80 (RF model slightly better), Brier scores between 0.09-0.17 for the RF and between 0.13-0.23 for the other two.
So based on the Brier scores it seems that the RF models has a slight advantage but I am wondering:
-Should I do more performance measurements? Which ones and why?
-How to interpret my results? My understanding is that there seems to be a linear association between the predictors as the GLM model performs well, but still the RF has a slight advantage in performance and accuracy but has the disadvantage of being a more "complicated" model.
I plan to do external validation with a different dataset but as of now I would be very interested in understanding if other measurements could shed some light on advantages of the different models and also I am sure I am missing something as I am new to the field and would be very interested to hear any advice/ opinions.
Thanks!