0
$\begingroup$

I have a dataset with 95% false and 5% true labels, some 200000 samples overall, I'm fitting a LightGBM model. I mainly need to focus on high precision and have low number of false positives, I don't care for accuracy much.

I have tried playing around with decision boundary after fitting and increasing weight for the positive class, this helps but I wonder if there is something else I could do.

Because the dataset is very unbalanced I think that the model is spending a lot of effort on the TN/FN boundary which I really don't care about. Also my intuition is that the standard cross-entropy loss is implicitly more focused on accuracy rather than precision.

I wonder if I could perhaps somehow pre-filter my dataset to maybe throw away 50% samples, but increase the initial T/F ratio. Or maybe this is what LightGBM already does and what I want is fundamentally impossible. Or perhaps there is an alternative to cross-entropy loss that I could use.

Improve this question
$\endgroup$
4
  • $\begingroup$ Why not always predict the false class? Then you never have to worry about predicting a true when the right answer is false! $\endgroup$
    – Dave
    Commented Sep 23, 2023 at 20:04
  • $\begingroup$ @Dave I don't follow, are you being facetious? Predicting the false class to achieve high precision on false is great, but I obviously care about the true class. Having zero recall on true is not helpful, even though the prediction is technically high precision. $\endgroup$
    – Fireant
    Commented Sep 23, 2023 at 20:11
  • $\begingroup$ I mainly need to focus on high precision and have low number of false positives, I don't care for accuracy much. This makes it sound like my suggestion is exactly what you need to do: never worry about a false positive ever again (not quite the same as high precision, sure). If that’s not what you meant, perhaps you can clarify why. Do you have a particular goal in mind or some kind of cost associated with the mistakes that makes you more reluctant to have false positives than false negatives? Your comment makes it sound like you might. $\endgroup$
    – Dave
    Commented Sep 23, 2023 at 20:25
  • 1
    $\begingroup$ If my intention was to completely reduce false positives above everything else, I'd mark the whole dataset as false rather than doing machine learning. I obviously want to find some true positives. I'm taking actions based on the positive prediction and it is important that I don't take actions based on false positives. Missing and action does not cost me as much as taking a wrong one. $\endgroup$
    – Fireant
    Commented Sep 23, 2023 at 20:45

1 Answer 1

Reset to default
1
$\begingroup$

During training you can place "more emphasis" on a given class (or sample) by either:

  • class weights to weight the error associated to a given class. There are usually used to balance the error associated to imbalanced classes. From the doc (here) there is a class_weight argument you can pass (assuming you're using the scikit-learn API in Python.)
  • Otherwise you can oversample the minority class (so duplicating the data) to balance the data.

Another approach - but this one after training - is to sort of calibrate the model by using validation data. Assuming the model predicts in a continuous [0, 1] range, you can tune a threshold $t$ such that a prediction $p$ is considered positive if $p>t$, therefore achieving a given precision or recall rate, e.g. $90\%$ precision, according to the chosen value of $t$. In this way you can also account for imbalance by trading-off errors, indeed.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.