Random Forest Model Giving Same Accuracy for different feature sets after tuning

Question

I am having this weird issue and cannot seem to find a solution. I am trying to tune a different random forest model for every different feature-set. Basically from a given data set, I have created 3 different feature sets. In first case you have all features, in second case you have slightly less number of features, and finally for the 3rd feature set you just have 5 features. I am trying to tune the model with RandomizedSearchCV.

param_grid = {'n_estimators' : range(1,100),
'max_depth' : np.linspace(1, 50, 5, endpoint=True),
'min_samples_split' : np.linspace(0.1, 1.0, 10, endpoint=True),
'min_samples_leaf' : np.linspace(0.1, 0.5, 5, endpoint=True),
'max_features' : range(1,X.shape[1]),
"criterion": ["gini","entropy"],}

grid = RandomizedSearchCV(clf_rf, param_distributions = param_grid, cv = 5, scoring = 'roc_auc', verbose=True, n_jobs=-1, random_state=1, return_train_score=False)
grid.fit(X,y)

Next, I use the grid.best_estimator_, the best model found by RandomizedSearchCV, to find to find the cross validated score.

estimator_cv = grid.best_estimator_
cv_results = cross_validate(estimator_cv, X, y, cv=10)
cv_accuracy = cv_results['test_score'].mean()
cv_accuracy

The issue I am facing is that after following the above steps for the the 3 different feature-sets, I am getting the same cross validated accuracy score of 0.8873372903702229. The roc_auc however is changing.

Answer 1

The fact that their accuracy is the same but their AUC score is different is encouraging. First and foremost, it would be good for you to inspect what is learned by the various trees. A good way to do that is to follow the instructions on that link.

What I think is that all 3 algorithms essentially learned that the same features are useful and that these features are present in each feature set. That is what random forests do, given a lot of features, they inherently "decide" which features are worth using.