I'm dealing with an imbalanced dataset and as usual it's very easy to obtain a high accuracy, but the recall on the less frequent class is very low. I would like to improve on the false negative of the less frequent class.

Focusing for concreteness on the sklearn Random Forest, one possible strategy is to set a class_weight penalizing the errors on the less frequent class and scoring with a sklearn scoring function as ROC.

grid = GridSearchCV(pipe, cv=number_of_cross_validations, n_jobs=1, param_grid=param_grid , scoring = scorer)
classifier = grid.fit(X_train, y_train)

where pipe is pipeline containing the RandomForestClassifier and param_grid contains different class_weight to try.

My question is probably related to this question, indeed class_weight alone seems to not be enough to lower significantly the false negative. As an extreme example, if I set:

 class_weight = {0: 0.0000001, 1: 0.9999999} 

(where 1 is the class with less instances, with a ratio 1:50), I would expect a final classifier predicting nearly always 1, since every time it makes an error the cost to pay is very high. But that's not what happen, the fitted classifier continue to make a significant number of false negative. Why is it so?

This is most likely because sklearn has chosen your majority class for you and you simply need to weigh your classes the other way around with "0" being the majority class.

If your classes don't have a class overlap problem, you might want to try some re-sampling techniques. you can oversample the class with less records, or look into packages like SMOTE ,ROSE.

Good example for SMOTE: https://www.kaggle.com/qianchao/smote-with-imbalance-data#

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.