2
$\begingroup$

I am using an imbalanced dataset (rare positive cases) to learn models for prediction and the final good AUC is 0.92 but the F1 score is very low0.2.

Is it possible to add some key features which will change the class probabilistic distribution and thus we can get a threshold to generate a higher F1?

Here is an example:

In my original model, I get a class probabilistic distributions shown below: enter image description here

I can adjust the threshold to make a better precision but meanwhile cut some recall off. It is due to the large overlapping area between two distributions.

Then I use an extreme dataset, i.e. include the target itself as a feature to learn. As a result, I can see I split the distribution completely disjointed. enter image description here

Dose it mean if I introduce a strong feature, I can split the distribution to some extent and thus promote the precision and thus f1 score? Or please advise how to improve precision under imbalanced classification issue.

Many thanks

$\endgroup$
  • $\begingroup$ Sorry, I am bit confused, what do you mean by key features? Feature Selection & Reduction may improve the results. Various Classifiers have parameters to adjust for imbalance; this would be your case to promote the undersampled Class and the two distributions perhaps. Moreover, you can use Sampling techniques (Undersampling, Oversampling) as proposed. The Precision/Recall is an unfair tradeoff if you have an Undersampled Class, increasing the threshold to its favour would usually drastically lower the Precision % and somewhat increase the Recall%. $\endgroup$ – Grzegorz Jun 9 '17 at 16:45
  • $\begingroup$ Yes I will try more feature selection even though I have try the univariate method and informative method. Besides I will add more potentially important features to see how it work. $\endgroup$ – LUSAQX Jun 9 '17 at 22:46
1
$\begingroup$

Introducing a Strong Feature would definitely help as it is "Strong" :-) .. if you do not have a sure-success feature then you may try changing penalty of miss-classification to start with.

You may try synthetic (i.e. SMOTE) or non-synthetic (domain based) approaches to bulk up the lower class.

Also, if this is a very very rare class the repeated sampling techniques may work

| improve this answer | |
$\endgroup$
  • $\begingroup$ Try this for implementation of SMOTE and other methods for tackling unbalanced data. $\endgroup$ – janpreet singh Jun 9 '17 at 13:19
  • $\begingroup$ Thank you two very much. My problem is probably not in the resampling for imbalance and I had done that for model training. My problem is given a good model with high AUC, if it face a imbalanced test set, it would generate too many FP cases and thus low precision and f1 score. $\endgroup$ – LUSAQX Jun 9 '17 at 22:43
  • $\begingroup$ And my first priority concern is the second part of my question presented in the last sentence of the post: how to improve precision under imbalanced classification issue. $\endgroup$ – LUSAQX Jun 9 '17 at 22:48
  • $\begingroup$ To my knowledge, ensemble method is robust on imbalanced data and the re-sampling should not contribute too much with these algorithms. $\endgroup$ – LUSAQX Jun 9 '17 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.