0
$\begingroup$

I have an imbalanced data where the ratio between positive to negative samples is 1:3 (positive samples are 3 times higher than negative). For my case it is is important to have a higher precision (and lower FPR) even if it comes at the cost of low recall (higher FNs). I intend to reach this goal by training a random forest with a class weight of negative class many times more than the positive class and observe that precision increases (lesser FPs) as I increase the weight of negative class and recall drops (more FNs). This was kind of what I expected. See a sample table below:

enter image description here

Next I also try to see if I can also train the model using an intentionally imbalanced data to introduce a bias towards classification of negative class. I try to achieve this by under-sampling my positive class with different fractions. So 1:10 means my negative samples are ten times more than positive in the training phase. What I observe now is that both the precision and recall go lower as I keep decreasing the number of positive samples in the training (hence making the negative class majority). Why is that Precision drops in this case although the FPR is decreasing? Should precision and FPR not be inversely proportional? Thanks

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ In which mode do you calculate metrics: micro, macro, weighted? $\endgroup$
    – Piotr Rarus
    Commented Dec 10, 2019 at 13:55
  • $\begingroup$ My problem is binary so I do not define this parameter. $\endgroup$
    – David
    Commented Dec 10, 2019 at 14:20
  • $\begingroup$ How big is your dataset? It may be that the drastic undersampling just loses too much information. Also, I don't quite understand your setup: if you are undersampling the negative class that originally was 1/4 of your data, how do you get to "negative samples are ten times more"? $\endgroup$
    – Ben Reiniger ♦
    Commented Dec 10, 2019 at 16:00
  • $\begingroup$ Sorry I meant undersampling of positive class (not negative. The dataset is quite big to start with (over 700k samples where majority class if positive samples) $\endgroup$
    – David
    Commented Dec 10, 2019 at 17:00
  • $\begingroup$ cross-posted at stats.stackexchange.com/q/440173/232706 $\endgroup$
    – Ben Reiniger ♦
    Commented Jan 2, 2020 at 20:30

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.