I am trying to learn more about Feature Selection in machine learning. I am working on a dataset that contains 17 features, and I have achieved about 75% accuracy on a Random Forest model with no parameters by manually selecting about 8 features using a correlation matrix by leaving those that has the same correlation score and select the features that I think should be used to train the model. I tried to implement RFE with Cross Validation, but it returns different feature every time I run the notebook. It also returns different features that I manually picked which I think is the reason why my model returns a lower accuracy score. My code is as follows:

from sklearn.feature_selection import RFECV
from sklearn.model_selection import StratifiedKFold

min_features_to_select = 1  # Minimum number of features to consider
clf = RandomForestClassifier(n_estimators=500, random_state=42) # I am using a RF Classifier
cv = StratifiedKFold(n_splits = 5, shuffle=False)

rfecv = RFECV(
rfecv.fit(X, y)

print(f"Optimal number of features: {rfecv.n_features_}")
print(f"Selected features are {X.columns[rfecv.support_]}")

I took this from the RFECV example in the sklearn documentation. I am really trying to figure out if there is a systematic way to pick features rather than just manually selecting them using correlation matrix.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.