0
$\begingroup$

I am trying to build a classification xgboost model at work, and I'm potentially facing overfitting issue that I have never seen before.

  • My training sample size is 320,000 X 718 and testing sample is 80,000 X 78 (after doing 80-20 split)
  • Features are a mix of continuous and one-hot encoded variables
  • Event vs Non-Event is 50%-50% (for both training and testing)
  • At the end of the day, my training accuracy is 98.07% (clearly overfitting), but my testing accuracy is also around 98.05% (testing also has 50-50% event vs non-event)
  • Unseen data is performing well in terms of accuracy, but its huge value seems unreal to me.

I had completed the following steps for data preparation and model evaluation.:

  1. replacing NULL continuous values with 0

  2. removing features having correlation > 0.5 (this reduced features from some 2000+ to 718)

  3. hypertuned using below parameters using 5 fold cross validation: lr = [0.01,0.05,0.1,0.2], ne = [200], md = [3,4,5]

  4. important parts of my model fit:

    train_X, test_X, train_y, test_y = train_test_split(X, y, test_size = 0.2, random_state=25)
    
    xgboost = XGBClassifier(subsample = 0.8, # subsample = 0.8 ideal for big datasets
            silent=False,  # whether print messages during construction
            colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
            gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
            objective='binary:logistic',
            eval_metric = ["error"]
          )
    
    clf = GridSearchCV(xgboost,{
    'learning_rate':lr,
    'n_estimators':ne,
    'max_depth':md
    },cv = 5,return_train_score = False)
    
    
    xgboost_ht = XGBClassifier(
            learning_rate = 0.2, # shrinkage for updating the rules
            max_depth = 5, # maximum tree depth for base learners
            n_estimators = 200, # number of boosting rounds
            subsample = 0.8, # subsample = 0.8 ideal for big datasets
            silent=False,  # whether print messages during construction
            colsample_bytree = 0.4, # subsample ratio of columns when constructing each tree
            gamma=10, # minimum loss reduction required to make a further partition on a leaf node of the tree, regularisation parameter
            objective='binary:logistic',
            eval_metric = ["error"]
          )
    
    
    xgboost_ht.fit(train_X,train_y)
    y_pred = xgboost_ht.predict(test_X)
    accuracy_score(y_true = test_y,y_pred = y_pred)
    0.980775
    

I can't comprehend under what scenarios even an unseen test dataset would exhibit higher accuracy. I have normally seen test to perform lower than train, with accuracies in the range 70%-80%.

PS - During data preparation, I had made the sample 50%-50% because originally the event proportion is 0.05%. So this is an imbalanced classification problem where I oversampled.

$\endgroup$
2
  • 1
    $\begingroup$ 1) I would recommend that you stop oversampling, as class imbalance (even considerable imbalance like you have) is rarely a problem. // 2) When you say that you typically observe $70\%$ or $80\%$ accuracy from other models, how (if at all) do they handle the class imbalance? // 3) It’s good to be skeptical about performance that sounds too good to be true, but if you’ve been rigorous with your work, the goal is strong performance, so celebrating might be warranted! $\endgroup$
    – Dave
    Nov 18, 2022 at 17:50
  • $\begingroup$ What cutoff value is used in your code? Do you know? Is this the optimal cutoff value for the problem you are solving (benefits of TP and TN vs costs of FP and FN). Accuracy is am improper scoring rule. If the cutoff value is not optimized to your problem you might not be looking at the appropriate metric - fharrell.com/post/class-damage and there are many other articles. $\endgroup$
    – Craig
    Nov 18, 2022 at 19:12

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.