My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of >100k samples. Classifying foreground class as background is quite OK, while classifying background as foreground should be penalised.
I am using Scikit-learn's RandomForests, and I was experimenting with SVM and OneVsRest classifiers as well. I would like to specify the scoring metrics used for the method
fit() of the model, so it will correspond to my goal (I imagine something like fitness function with evolution algorithms). However, API does not allow something like that.
So far I tried:
class_weightparameter of the model. If I set it so it represents the real world, then the classifier learns to classify everything as background having accuracy >99 %. If I set
class_weight = 'balanced', then it seems better, but it has high false positive rate.
- Use scoring method for
GridSearchCV, which outputs values specified by me (even F1-score makes more sense than simple accuracy), but it is used only for the parameter selection and the final model is learnt by
fit()method, which again ignores my scoring.
Is there a way to provide my own custom scoring function for the