Random Forest Classifier cannot recognise parameter grid

Question

I am trying to run the below code to extract the feature importances of my random forest, but I'm getting the following error TypeError: init() got an unexpected keyword argument 'randomforestclassifier__max_depth'. Can anyone tell me what is wrong?

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from imblearn.over_sampling import RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.metrics import f1_score

x, y = make_classification(n_samples=10000, weights=[0.99], flip_y=0)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 1)

paramgrid_rf = {'n_estimators': [500],
                'max_depth': [4],
                'random_state': [0],
                'max_features': ['sqrt']
                }

imba_pipeline_rf = make_pipeline(RandomOverSampler(sampling_strategy=0.35, random_state=0),
                             RandomUnderSampler(sampling_strategy=0.9, random_state=0),
                             RandomForestClassifier())

new_params = {'randomforestclassifier__' + key: paramgrid_rf[key] for key in paramgrid_rf}
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
grid_imba_rf = GridSearchCV(imba_pipeline_rf, param_grid=new_params, cv=cv, scoring='f1',
                        return_train_score=True)
scores = cross_val_score(imba_pipeline_rf, x_train, y_train, scoring='f1', cv=cv, n_jobs=-1)
grid_imba_rf.fit(x_train, y_train)
y_pred_rf = grid_imba_rf.predict(x_test)
print('F1 score on validation data: ', f1_score(y_test, y_pred_rf))

rf_final = RandomForestClassifier(**grid_imba_rf.best_params_).fit(x_train_res,y_train_res)
rf_final.feature_importances_

Answer 1

Your grid search dictionary contains the argument names with the pipeline step name in front of it, i.e. 'randomforestclassifier__max_depth'. Instead, the RandomForestClassifier has argument names without the pipeline step name, i.e. max_depth. You therefore need to remove the first part of the string which denotes the name of the step in your original pipeline. You can do this using a dictionary comprehension:

# original
{'randomforestclassifier__max_depth': 4, 'randomforestclassifier__max_features': 'sqrt', 'randomforestclassifier__n_estimators': 500, 'randomforestclassifier__random_state': 0}

# splitting the key on '__' and take only the last part
{k.split("__")[-1]: v for k, v in grid_imba_rf.best_params_.items()}
# {'max_depth': 4, 'max_features': 'sqrt', 'n_estimators': 500, 'random_state': 0}

This changes one line in the original script to:

rf_final = RandomForestClassifier(
    **{k.split("__")[-1]: v for k, v in grid_imba_rf.best_params_.items()}
).fit(x_train,y_train)
rf_final.feature_importances_

Answer 2

Welcome to the community. By taking a quick look at your code, it seems to be that your RandomForestClassifier instance is receiving randomforestclassifier__max_depth as input param, instead of just the sklearn defined param name max_depth.

The error seems to come from your definition of new_params when adding 'randomforestclassifier__'.

Make sure you pass to the RF classifier the defined params with the correct names.