1
$\begingroup$

I have a classifier with a heavily imbalanced dataset (1000 of each negative label for each positive.)

I'm running a GradientBoostingClassifier with moderate success (AUC .75) but the curve has this strange look:

enter image description here

Any good ideas on what would cause the curve to have this behaviour?

$\endgroup$
3
$\begingroup$

Davis and Goadrich have explained the relationship between ROC and PR Curves in their paper. It is always recommended to use PR curve over the ROC curve in the presence of highly imbalanced data.

Back to the behavior of your ROC curve, It seems that you don't have more threshold points! I would also agree with Dan and do K-fold CV.

Davis, J. and Goadrich, M., 2006, June. The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd international conference on Machine learning (pp. 233-240). ACM.

$\endgroup$
2
$\begingroup$

I think there's some predictor Q for one or a few of your positive examples that also applies to a lot of negative examples. Because you have so few positive examples, there's not much to separate the good from the mediocre predictors for them. When you got to the validation set, Q must have applied to a greater proportion of negative examples than it did in the training set.

To mitigate this, try n-fold cross-validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.