Select more or less features if results are almost the same

Question

I am having a dataset of 3500 observations with 70 features each with binary labels/targets for classifications purposes.

My aim is to score more than 90% precision and the highest recall possible for this precision.

I have tested many algorithms and thus far Random Forest performs the best.

Specifically, with 70 features I am getting:

• Auroc: 0.71
• Precision: 0.94
• Recall: 0.18

However, I had a look at the importance of my features by calling the built-in function feature_importances_ of the RandomForestClassifier of SkLearn and by also using the Boruta algorithm with the boruta_py module.

I removed the 20 least important features. The number 20 was quite arbitrary because even the next 10 least important features were relatively close in terms of importance to these 20 least important ones. Only the top 10 most important features had a significant difference in importance in comparison with the others.

Then the results with 50 features were the following:

• Auroc: 0.7
• Precision: 0.91
• Recall: 0.17

Since I can get almost the same results with less features should I remove these features?

Please also into take into account that in the future my dataset will be expanded to 5000 observations.

Answer 1

It is very contextual. There's a rule that feeding all the data will yield better (unbiased) results but not at all times. If you know anything about the data, then it is always advised to remove the most obvious ones that won't matter. Additionally, you can use other metrics like 'P-Value' to check the variable significance.

Since there's more data that will be added in the future, I'd rather wait for it and then remove the insignificant variables.