1
$\begingroup$

The models were evaluated using 10-fold cross validation.

foldCount = StratifiedKFold(10, shuffle=True, random_state=1)

The models in question are XGBoost.

xgb = XGBClassifier(verbosity=2, random_state=0, n_estimators=100, max_depth=10, learning_rate=0.35, gpu_id=0, tree_method='gpu_hist', predictor='gpu_predictor')

The shape of the dataset is (117k, 34) after preprocessing and feature selection.

The dataset was balanced using imblearn's SMOTEENN.

from imblearn.combine import SMOTEENN
smote_enn = SMOTEENN(random_state=42, sampling_strategy = 'not majority')
X_normalized, y = smote_enn.fit_resample(X_normalized, y)

Data Before Balancing + Model Performance

enter image description here Blockquote

Data After Balancing (with SMOTE-ENN) + Model Performance

Blockquote Blockquote

Improve this question
$\endgroup$
2
  • $\begingroup$ Have you applied the SMOTE to both the train and test data? The code makes it looks like you ran SMOTE, split the balanced data, and then trained your model. // Are you sure there is a problem to solve? Class imbalance being a problem is largely disputed by statistics. $\endgroup$
    – Dave
    Mar 20 at 14:13
  • $\begingroup$ Hey, Dave. I've applied it to all of the data (I'm using K-fold cross validation). I balanced the dataset with SMOTE-ENN and then proceeded to train the model across 10 folds. These are the results averaged across the 10 folds. Also, I find your last comment intriguing. Can you please elaborate as to why class imbalance is disputed by statistics? This is news to me. $\endgroup$
    – Tariq
    Mar 20 at 14:42

1 Answer 1

Reset to default
1
$\begingroup$

While SMOTE is usually a technique that aims to solve what turns out to be mostly a non-problem and probably is not even good at what it claims to do, applying such a technique to the holdout data is the worst idea of all. The point of a technique like SMOTE is to improve the performance of a model that is in production, where you probably do have the class imbalance. Then the point of having a test set is to have some sense of how your model will perform when it is used for real. If your test set has artificial balancing, you are deceiving yourself by evaluating performance on that set. Of course the model is more likely to predict minority classes, but those minority classes really are not so common, so you do not want to predict them as often as you predict the majority class.

Consequently, to get an honest view of your model performance (including a check of overfitting), you have to make predictions on honest data. Select your training data, SMOTE the training data if you feel you have to (despite protests from the statisticians), and then evaluate on a data set that reflects the real conditions where your model would be operating.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much, Dave. Slightly embarrassed, but I do sincerely appreciate the lesson. I'm still learning! I'll try again with SMOTE-ENN only on the training set and see if it performs well on the untouched test set. The reason I've been exploring data balancing techniques is because I can't seem to improve the F1 score on this dataset no matter what I do. $\endgroup$
    – Tariq
    Mar 20 at 15:12
  • 1
    $\begingroup$ @tuh940 That you are interested in hard classifications instead of predicted probabilities likely represents another mistake. The first link gets into why that is the case. // No need for embarassment. Everyone who has become an expert started out as a beginner. $\endgroup$
    – Dave
    Mar 20 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.