hope to find you well ! I am trying to build a model to classiffy customers with propensity to buy, but i cannot get rid of overfitting! My approach is the following: I have created the train dataset with unbalanced approach and have now a target 1 of 6% and a total of 6.755 rows and 252 columns. On the other hand, the test dataset has 313.587 rows and target 1 is only 34 of the cases (really low %). The test set was constructed to reflect the reality : the universe to score is all customers from one month, so for the test set I also chose all customers from one month and actually this is a product with low expression... I used autoML code from H2o in R and I am getting very different AUC between train (0.9) and test (0.6). The code i used is the one below:

aml <- h2o.automl(x = predictors, y = response,
                  training_frame = train_h2o,
                  max_models = 10,
                  seed = 3)

Any advice on what I can do to prevent from overfitting?

  • $\begingroup$ Welcome to DataScienceSE. The problem might not be overfitting, at least not in a strict sense: there is overfitting in a supervised model if the model performance is significantly lower in the test set than in the training set, but under the assumption that the test set follows the same distribution as the training set. Here the assumption is not satisfied, I understand that it's what the problem looks like in reality but that makes it a poor candidate for a proper supervised learning problem. $\endgroup$
    – Erwan
    Apr 11 at 10:56
  • $\begingroup$ Hi! Thanks for the answer! Actually that rises a question : what about when we use sampling methods ? The distribution is not the same also.. $\endgroup$ Apr 12 at 20:18
  • $\begingroup$ Resampling the training set is indeed a bias introduced on purpose to force the model to take care of the minority class. If used excessively resampling could cause bad results similar to what you observe. However in your case the difference in the distributions is arbitrary, there's no "calculated reason" why it's in this particular way. I'm not knowledgeable about these but I know that there are methods for 'adapting' a model to a new dataset, it's part of semi-supervised methods I think. I would assume that this might be a good direction to study for your use case. $\endgroup$
    – Erwan
    Apr 12 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.