Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is expected to be low. How to choose a good decision boundary for such cases.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You don't necessarily have to train any differently if you are happy with the result. If you aren't, undersample the majority class and/or adjust the class weights in your loss function. The thing you really have to be more careful about is the evaluation of the result because a naive metric like the accuracy would give you false hope since the imbalance allows you to achieve a high accuracy by simply always predicting the majority class or randomly predicting each class according to its probability of occurrence. It is this baseline against which you have to judge your success; like the coefficient of determination in regression, where the baseline is the sample mean.
You could use a graphical approach (ROC or precision/recall curves), or report a size-weighted average of the class positive rate instead of the accuracy. Here are some other ideas: Comparison of Evaluation Metrics in Classification Applications with Imbalanced Datasets.
$\begingroup$
$\endgroup$
Check whether the classifier offers class_weight='balanced' or 'auto' option . I had better results as when down or up-sampled. Best classifier for me turned out to be SGD from sklearn. Auto-sklearn shows even better results.
