I have been dealing with a classification problem. Real issue is the imbalance here

I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification Personally I am not a big fan of oversampling/under-sampling since they mess the distribution up. I also tried stratified sampling so that same proportion is kept while training but its not working. For my present approach I take a sample of -ve points (say 100,000) and then train with 300 +ve samples and getting an ROC-AUC of ~85 but almost no case predicted reaches probability >51% (after using model.predict_proba). Any tips on handling such an extreme imbalance? additionally I would only like to stay with tree based or boosting models since they provide interpretability (hence avoiding neural network, anomaly detection , autoencoder for now). Any hints, resources appreciated! Below is the code I am using.

df = read_data(path)
Pkl_Filename = path + "/catboost.pkl"
# if not os.path.exists(Pkl_Filename):
X = df.drop(ep_config['target_field'], axis=1)
y = df[ep_config['target_field']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, stratify=y)
clf = CatBoostClassifier(eval_metric='Recall', use_best_model=True, scale_pos_weight=...)
sm = SMOTE(random_state=27, ratio=1.0)
X_train, y_train = sm.fit_sample(X_train, y_train)
smote = clf.fit(X_train, y_train, eval_set=(X_test, y_test))

# smote_pred = smote.predict(X_test)

# Checking accuracy
# accuracy_score(y_test, smote_pred)
# clf.fit(X_train, y_train, eval_set=(X_test, y_test))
with open(Pkl_Filename, 'wb') as file:
    pickle.dump(smote, file)
# else:
#     clf = pickle.loads(Pkl_Filename)
logging.info('the test roc is :{:.6f}'.format(roc_auc_score(y_test, smote.predict_proba(X_test)[:,1])))

have you already tried using only very little of the -ve cases? So for example to train your model on 900 points total, 600/300? Then stratified sampling should still work fine. Then I'd evaluate your model based on it's ability to predict -ve cases and just monitor the performance it get's on the (in your case) gigantic test dataset that the model hasn't seen before.

Also, you could generate "synthetic" +ve cases by bootstrapping from the real data to see how that influences the model. Similar to what is described here: https://sci2s.ugr.es/keel/pdf/specific/articulo/dupret_science.pdf

  • $\begingroup$ as I said , we are using training on very little -ve cases for example 10,000 -ve and 300 +ve (along with class weights of course) . so this thing is not working good. by not good i mean ~65 roc auc and no positive predicted class has probability >51%. by generating synthetic samples you mean oversampling/undersampling/smote right ? $\endgroup$
    – billie_joe
    Feb 19 at 9:54
  • $\begingroup$ I think I'd go a lot lower with your sample number first. You can always add more data later since you already have it. Let's say you're doing stratified k-fold with k=10. That'd roughly give you 27 +ve in your training set which is probably not enough for the model to actually "learn it" in the imho. So maybe randomly pick -ve a couple times and try with very little data first. Decision trees should still perform reasonably well. Yeah, SMOTE would be sth you could try $\endgroup$
    – ttreis
    Feb 19 at 10:32
  • $\begingroup$ so you are saying to pick around 1000 -ve and +ve and then train ?(using stratified sampling of course) and after that smote? $\endgroup$
    – billie_joe
    Feb 19 at 11:26
  • $\begingroup$ I would try two approaches: First, pick around 1000 -ve and your 300 +ve, train a cross-validated model and then see how it performs. If it's not garbage, see how it performs on the remaining -ve data points. You could even try a k-fold approach in which every new -ve set is truly new and randomly sampled from the whole data. The other approach would be to use SMOTE to generate synthetic +ve cases based on the existing one and then see how a model on that performs. But for this I also wouldn't take ALL of your -ve cases. $\endgroup$
    – ttreis
    Feb 19 at 11:35
  • $\begingroup$ "But for this I also wouldn't take ALL of your -ve cases" why? oh ok.so essentially split the data first into train and test and do oversampling only on train and leave test untouched right $\endgroup$
    – billie_joe
    Feb 19 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.