2
$\begingroup$

I am using an imbalanced dataset (rare positive cases) to learn models for prediction and the final good AUC is 0.92 but the F1 score is very low0.2.

Is it possible to add some key features which will change the class probabilistic distribution and thus we can get a threshold to generate a higher F1?

Here is an example:

In my original model, I get a class probabilistic distributions shown below: enter image description here

I can adjust the threshold to make a better precision but meanwhile cut some recall off. It is due to the large overlapping area between two distributions.

Then I use an extreme dataset, i.e. include the target itself as a feature to learn. As a result, I can see I split the distribution completely disjointed. enter image description here

Dose it mean if I introduce a strong feature, I can split the distribution to some extent and thus promote the precision and thus f1 score? Or please advise how to improve precision under imbalanced classification issue.

Many thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ Sorry, I am bit confused, what do you mean by key features? Feature Selection & Reduction may improve the results. Various Classifiers have parameters to adjust for imbalance; this would be your case to promote the undersampled Class and the two distributions perhaps. Moreover, you can use Sampling techniques (Undersampling, Oversampling) as proposed. The Precision/Recall is an unfair tradeoff if you have an Undersampled Class, increasing the threshold to its favour would usually drastically lower the Precision % and somewhat increase the Recall%. $\endgroup$
    – 20-roso
    Commented Jun 9, 2017 at 16:45
  • $\begingroup$ Yes I will try more feature selection even though I have try the univariate method and informative method. Besides I will add more potentially important features to see how it work. $\endgroup$
    – LUSAQX
    Commented Jun 9, 2017 at 22:46

1 Answer 1

Reset to default
1
$\begingroup$

Introducing a Strong Feature would definitely help as it is "Strong" :-) .. if you do not have a sure-success feature then you may try changing penalty of miss-classification to start with.

You may try synthetic (i.e. SMOTE) or non-synthetic (domain based) approaches to bulk up the lower class.

Also, if this is a very very rare class the repeated sampling techniques may work

Improve this answer
$\endgroup$
4
  • $\begingroup$ Try this for implementation of SMOTE and other methods for tackling unbalanced data. $\endgroup$
    – janpreet singh
    Commented Jun 9, 2017 at 13:19
  • $\begingroup$ Thank you two very much. My problem is probably not in the resampling for imbalance and I had done that for model training. My problem is given a good model with high AUC, if it face a imbalanced test set, it would generate too many FP cases and thus low precision and f1 score. $\endgroup$
    – LUSAQX
    Commented Jun 9, 2017 at 22:43
  • $\begingroup$ And my first priority concern is the second part of my question presented in the last sentence of the post: how to improve precision under imbalanced classification issue. $\endgroup$
    – LUSAQX
    Commented Jun 9, 2017 at 22:48
  • $\begingroup$ To my knowledge, ensemble method is robust on imbalanced data and the re-sampling should not contribute too much with these algorithms. $\endgroup$
    – LUSAQX
    Commented Jun 9, 2017 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.