1
$\begingroup$

Extreme values are predicted by my trained xgboost classification model in BQML for both events (Y=1) and non-events (Y=0). For all event observations, the model calculates probability scores that are almost 0.98 and above, while for all non-event data, they are almost 0.02. The box-plots below show the distribution of probability scores for these observations of events and non-events. Furthermore, this comes from the testing dataset. This is overfitting. Any suggestions for how to approach this?

enter image description here

$\endgroup$
1
  • $\begingroup$ There is not enough details to offer any appropriate solution, also there is little information on what you have done to resolve your issue. As a generic advice, try to simplify the problem and then slowly make it more complex. For example, if you fed your label as a feature, you would definitely expect classifier to work well. Does it? Next try to predict label from just one feature. This should give you a hint on which features are important and which are less so. Next, if you combine several important features, you would not expect classifier performance to drop etc... $\endgroup$
    – Cryo
    Commented Oct 8, 2023 at 18:17

2 Answers 2

3
$\begingroup$

This is not necessarily overfitting, but it may indicate data leakage i.e you are passing information to the model that is not supposed to be there it may be:

  1. Information that is generated after the event of the target. For example, credit risk would be using information on credit accounts opened after the moment you are evaluating the customer.
  2. The target itself. For example, you add a feature generated only when the target equals 0 or 1. Imagine, for example, using a null feature when the customer is not approved and has a value different from null otherwise.

So it is a problem that may be difficult to address, but you can run a simple validation using permutation importance.

If you see that there is one feature that, when dropped, has a huge impact, it is likely that feature has leakage and thus, the metrics you are seeing are not trustable.

$\endgroup$
1
$\begingroup$

A high score on the test set does not indicate overfitting.

See Why 100% accuracy on test data is not good? ; you're not quite reaching perfect performance, but you're quite close, and in that seeing it is natural for the model to make confident predictions. You should focus on making sure you haven't leaked future information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.