2
$\begingroup$

all,

i am training light gradient boosting and have used all of the necessary parameters to help in over fitting.i plot the predicted probabilities (i..e probabililty has cancer) distribution from the model (after calibrating using calibrated classifier) i.e. their histogram or kde. as you can see from below the probabilities for my class 1 are concentrated on the upper and lower end.

i have tried playing around with bandwith too to smooth this a little and it doesn't smooth the bumps too much. what do you think this shows about my model? isn't it a good thing that the model for class 1 (which is has cancer) is assigning a greater probability for this class?

i am unsure how to interpret this or where i could be going wrong

enter image description here

the red curve is positive class (has cancer) and the blue curve is hasn't. below is plot used to generate.

results = df[['label','predicted_prob']]

colors = ['b', 'r']



for label in [0, 1]:
    results[results['label'] == label]['predicted_prob'].plot.kde(bw_method=0.35,color=colors[label])
plt.xlim(0,1)
Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ maybe the best way to check overfitting is to test on new labelled data no seen before and measure performance on that $\endgroup$
    – Nikos M.
    Jul 25, 2020 at 16:46

1 Answer 1

Reset to default
0
$\begingroup$

Such a plot doesn't really tell you much about overfitting.

First, check that your calibration has worked well; it's possible that an incorrect calibration has pushed the probabilities to the extremes. Otherwise, the distribution of probabilities being so extreme suggests the data just naturally separates into a segment of easy-to-detect cancers and the rest. Among the latter, it looks like you get reasonably good but not great rank-ordering of cases.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Exactly what I thought. My calibration curve has indeed pushed the probabilities to extreme but it still looks better than not calibrated. Is it bad that for.positive class it detects probability of having cancer greater than 0.8? I think it does no $\endgroup$
    – Maths12
    Jul 25, 2020 at 21:54
  • $\begingroup$ @Maths12 There's nothing inherently wrong with getting 0.8 scores. It could indicate you have a variable that's "cheating," future information. And if you want to use it as a probability, you'll care more about how good your calibration is, but if you just want to make predictions this isn't so important. $\endgroup$
    – Ben Reiniger ♦
    Jul 26, 2020 at 0:28
  • $\begingroup$ Many thanks. Yes so the curve above is from when I've used isotonic calibration, I do want to use the probability output so I'll have to check what is going wrong with calibration $\endgroup$
    – Maths12
    Jul 26, 2020 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.