I've build two identical rf_classifier and trained with two identical datasets but with 2 different target variable (the sell or not sell of two different specific products, one for each algorithm).

df has 16k rows of data x 36 columns, for both the cases well balanced between the two classes.

To me, both models are behaving well, the first one with 78% accuracy and the second one with 84%.

The problem is that the first one behaves like that the probability distributions of an observation to be 1 is quite well distribuited (i'm using predict_proba to create 3 different classes at the end, manually setting the thresholds), the second one instead, even if in the test set it reach 84% (so it predicts well both classes), it tends to predict pretty much everything as 1 when given completely new data, with a probability > 0.8.

I don't have a lot of correlation with the output. One column has 0.5, other 4 coulmns are around 0.2, 2 more columns 0.1, all of the other are below 0.1.

I don't understand why my algorithm is behaving like this and the business, after being very happy for the results of the first one that, after it was provided with 120.000 new obs (clients data), allowed us to focus on like 1000 clients predicted as 1 for the first product to sell, now are facing that almost all of the same 120.000 clients are predicted as 1 for the second product, and they are assuming that it's my fault that I didn't train the algorithm well.

I don't understand what I'm doing wrong.

I've tried different algorithms like xgboost or NN, but the results are pretty much the same.

Coding wise I think that everything is good, it's not my first experience, but it's the first time that I'm facing something like this, it seems like the algorithm was trained on imbalanced data that were almost all class 1, instead the classes are really well balanced in the training (and in the test as well).

  • 4
    $\begingroup$ If it's doing well in the test set but not the fresh data, I would start by checking whether the distribution of the new data differs significantly from that of the train/test sets: plot the histograms for each feature on each dataset and overlay them, maybe compute the population stability index or similar distribution-shift metric. $\endgroup$
    – Ben Reiniger
    May 15 at 13:38
  • $\begingroup$ It seems that you scaled the dataset while training the model, but when you tested it with a new dataset, you provided it unscaled data . Could you please check this ? $\endgroup$
    – Foxbat
    May 16 at 6:30
  • $\begingroup$ @Foxbat since I'm using a rf classifier, I have scaled nothing. I've also tried with scaling (standard) but it gave me the exact same results $\endgroup$ May 16 at 6:37
  • $\begingroup$ Have you checked the probabilities of predicting 1 in both your test dataset and your new dataset? Visualizing these probabilities might help identify any issues. Also, did you compare the distributions of your training dataset and the new dataset? Differences in their distributions could explain why the predictions are different $\endgroup$
    – Foxbat
    May 16 at 7:08
  • 1
    $\begingroup$ Another method could be to check the CalibratedClassifierCV from sklearn to ensure your model's accuracy. You can compare the CalibratedClassifierCV Brier score with your model's Brier score to check whether there are significant differences. $\endgroup$
    – Foxbat
    May 16 at 7:25


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.