7
$\begingroup$

My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of >100k samples. Classifying foreground class as background is quite OK, while classifying background as foreground should be penalised.

I am using Scikit-learn's RandomForests, and I was experimenting with SVM and OneVsRest classifiers as well. I would like to specify the scoring metrics used for the method fit() of the model, so it will correspond to my goal (I imagine something like fitness function with evolution algorithms). However, API does not allow something like that.

So far I tried:

  • Use class_weight parameter of the model. If I set it so it represents the real world, then the classifier learns to classify everything as background having accuracy >99 %. If I set class_weight = 'balanced', then it seems better, but it has high false positive rate.
  • Use scoring method for GridSearchCV, which outputs values specified by me (even F1-score makes more sense than simple accuracy), but it is used only for the parameter selection and the final model is learnt by fit() method, which again ignores my scoring.

Is there a way to provide my own custom scoring function for the fit() method?

Improve this question
$\endgroup$
1
  • $\begingroup$ When I run the fit() method twice on the same GridSearch object, changing the scoring= parameter each time, I get different results for the prediction on my test set (i.e. for recall, confusion matrix, etc...). This would suggest that when you change the scoring= param, this does actually affect which best_estimator is chosen. $\endgroup$
    – Brent Nixon
    Commented Feb 15, 2018 at 19:37

1 Answer 1

Reset to default
1
$\begingroup$

I think I understand what you are trying to do. First, let me attempt to obtain clarity on two concepts that may be getting conflated (if not for you, then for other users).

When a random forest is being fit, it typically uses entropy or gini impurity for categorical outcome variables, and mean squared error or mean absolute error for numerical outcome variables -- this is the model building step. However, when the fit model is being evaluated, the scoring method is used to understand the models performance -- this is the step where we learn how accurate the model might be.

I've seen that sklearn allows specifying a custom scoring method, but as far as I know, I don't believe you can specify your own criterion parameter in the RandomForestClassifier or RandomForestRegressor methods/APIs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.