I am having a dataset of 3500 observations with 70 features each with binary labels/targets for classifications purposes.
My aim is to score more than 90% precision and the highest recall possible for this precision.
I have tested many algorithms and thus far Random Forest performs the best.
Specifically, with 70 features I am getting:
- Auroc: 0.71
- Precision: 0.94
- Recall: 0.18
However, I had a look at the importance of my features by calling the built-in function
feature_importances_ of the
SkLearn and by also using the Boruta algorithm with the
I removed the 20 least important features. The number 20 was quite arbitrary because even the next 10 least important features were relatively close in terms of importance to these 20 least important ones. Only the top 10 most important features had a significant difference in importance in comparison with the others.
Then the results with 50 features were the following:
- Auroc: 0.7
- Precision: 0.91
- Recall: 0.17
Since I can get almost the same results with less features should I remove these features?
Please also into take into account that in the future my dataset will be expanded to 5000 observations.