3
$\begingroup$

I am working with a really imbalanced dataset ($\approx$ 1% of positive cases) for a classification problem. I know that class balancing is an important step in this scenario.

I have two questions:

  1. Considering that I don't want to put the 0/1 label, but just to order the record according to the output score (it is always a calibrated probability of being in the positive class), is it still a good idea to do class balancing or, considering the specific output required, it is useless?

    Basically, I do not care about the cut-off point, but I just sort the record in order to identify the one with a higher probability of being positives.

  2. Considering the really small percentage of positive cases, is it better to do over/under sampling? Is there any rule-of-thumb to decide the proportion of resampling?

Thank you in advance!

$\endgroup$
2
$\begingroup$

With such a heavy imbalance and two classes (it seems) you could treat this as more of an outlier detection problem. You should read up on models and algorithms in that direction!

If you go forward with a traditional classification you need to balance the data set, consider methods such as SMOTE.

Depending on the size of your data I would generally recommend downsampling the majority class which avoids producing "synthetic" cases but advanced methods such as SMOTE basically take care of this decision for you.

Can you elaborate what you mean with your first question as well? A classification algorithm needs 0/1 labels and therefore the output score cannot be ordered in the way that you mention. Some classification algorithms put out a probability score instead of predicted label so if this is what you mean I can tell you that the imbalance will still be a problem.

$\endgroup$
3
  • $\begingroup$ I have edited the question to provide more details about the second points. Let's consider that working on the output scores I can add the 0/1 label to each record using an ideal cut-off point (eg. 0.5), but it is useless for me since I just want to identify the top-ranked records in order to contact them (eg. to sell some products,...) $\endgroup$
    – A1010
    Jul 20 '20 at 9:46
  • $\begingroup$ As said in this case this changes nothing about my answer and you'd need to deal with the imbalances. $\endgroup$
    – Fnguyen
    Jul 20 '20 at 10:04
  • $\begingroup$ About what @Fnguyen mentions, it sounds logic to look at this problem as a "novelty" detection problem due to the extreme imbalance; nevertheless, remembreing it would be a semi-supervised learning problem; two possible algorithms for that are scikit-learn.org/stable/modules/svm.html#svm-outlier-detection and scikit-learn.org/stable/modules/generated/…, both having an anomaly score with which you can tune your desired threshold $\endgroup$
    – German C M
    Jul 31 '20 at 15:27
2
$\begingroup$

Referring to a previous answer and a blog post (which I'm aware is not that relevant since the data is more balanced than yours), I think that your first approach should be without handling imbalance, and if you're happy with the results, no need to work towards balanced solutions.

As in many ML topics, the best way is to try, I recommend you to adapt the experiment in the blog post to your data.

However, a more specific answer to your question:

  1. I think that balancing usually messes the calibration of your classifiers on your training data, so if you needed calibrated predictions I would advocate for not using balancing. If you don't care about calibration, it is not that bad to balance.
  2. Under-sampling is better than over-sampling in my experience. The amount of under or over-sampling can be a hyperparameter to tune.
$\endgroup$
1
  • $\begingroup$ thank you very much. Do you have any reference for how to choose if it is better to under or oversample? $\endgroup$
    – A1010
    Aug 3 '20 at 8:50
2
$\begingroup$

Some Python Sklearn models have this option : class_weight="balanced". By that, you specify to your algorithm your data are unbalanced, and it makes the changes by itself. You can try this on few models, I had a better result with this option than by using the Downsampling Majority Class technique in a same problem

$\endgroup$
3
  • $\begingroup$ yes, you are right. Using this approach, you are able only to take into account the class balancing but you are not considering the specific class weights (the cost of misclassification). What do you think about it? $\endgroup$
    – A1010
    Jul 31 '20 at 13:55
  • 1
    $\begingroup$ Does "cost of misclassification" only has an influence on evaluation metrics values, or on model itself ? In fact I'm almost never using accuracy as an evaluation metric, but I prefer to go with AUC and checking TP FP TN FN at each threshold, and it works pretty well in my case, but I honestly don't know the difference between balancing and weights $\endgroup$
    – BeamsAdept
    Jul 31 '20 at 14:13
  • 2
    $\begingroup$ the weights should impact also in the model itself, not only in the metrics. If you think about the tree-based algorithms, it is possible to evaluate the node impurity taking into account the miss-classification or the cost of miss-classification, so it directly impacts the model structure $\endgroup$
    – A1010
    Jul 31 '20 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.