I was working on a model with following process:

  1. Split to training/validation/test sets
  2. Try a series of different models like GBM, RF, Logistic Regressions
  3. Optimize hyper-params on them using GridSearchCV with roc_auc as scoring metric
  4. I applied Oversampling methods and was careful that only training folds were oversampled (Imbalance is around 99:1)
  5. Tested model on validation set

In this example, I'm showing the result of my Random Forest model with SMOTE oversampling, but the result holds for the other models as well. The AUC is decent at around 0.72 and the ROC looks like following:

enter image description here

However, the Precision-Recall curve looks horrible:

enter image description here

I'm not sure why I'm running into this issue, but would appreciate any insight or suggestions on how I could change my approach or refine it.

  • $\begingroup$ The graph is telling you that you have horrible performance for the minority class. You class imbalance is huge and oversampling it 100 times is not going to solve the problem, you need to consider a different approach. $\endgroup$ Sep 28 '18 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.