0
$\begingroup$

I am following this tutorial to apply threshold tuning using precision-recall curve for an imbalanced dataset

Within the tutorial, a no-skill model is defined as:

A no-skill model is represented by a horizontal line with a precision that is the ratio of positive examples in the dataset (e.g. TP / (TP + TN)), or 0.01 on our synthetic dataset. perfect skill classifier has full precision and recall with a dot in the top-right corner.

The no skill model is the plotted using this code:

no_skill = len(testy[testy==1]) / len(testy)
pyplot.plot([0,1], [no_skill,no_skill], linestyle='--', label='No Skill')

enter image description here

Within my dataset, the no-skill model would be plotted at precision 0.1, which is the ratio of positive examples in train_y.

But my question is: wouldn't a no-skill model in an imbalanced dataset automatically label all observations as being from the majority class? Seeing as precision = TP/(TP+FP), in this case TPs=0 making precision=0? So shouldn't the no-skill model be plotted at precision=0?

$\endgroup$

1 Answer 1

0
$\begingroup$

Assuming the dummy model yielding constant probability for all samples, there are two possible thresholds: either everything is classified as positive or negative. So starting point will be (1, positive_class_ratio) and the ending point (0, 1) (by convention, as precision is undefined with no positives).

Tutorial version is still reasonable as its AUC conforms with sklearn average precision score for this case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.