1
$\begingroup$

I am using the LGBM model for binary classification. My train and test accuracies are 87% & 82% respectively with cross-validation of 89%. ROC-AUC score of 81%. But when evaluating model performance on an external validation test that has not been seen before, the model gives a roc-auc of 41%. Can somebody suggest what should be done?

$\endgroup$
1
  • $\begingroup$ First thing I would do is go back to basics and see how the training/test/validation sets are alike or different. Use frequencies, summaries, across the different levels. there are so many things that could go wrong. $\endgroup$ May 9, 2022 at 20:04

1 Answer 1

2
$\begingroup$

First, an AUC less than 50% is terrible: it means that you get better performance by switching the positive and negative labels! So the model is doing worse than nothing on this data.

In general there seems to be some problem with the design of the task and/or its evaluation: either the original test set and/or the validation set is/are not a representative sample for the target problem.

In case the validation set is supposed to be representative, it means that the test set is not valid, possibly due to data leakage (it contains information from the training set). In this case the test set evaluation is meaningless, and the fact the validation performance is much lower means that there is overfitting: the model is capturing details which happen by chance in the training set, because it's too complex and/or the training set is not representative enough.

$\endgroup$
5
  • $\begingroup$ Hello Erwan! I checked my data again too. I actually have data of chemical molecules, and when I cross-checked my validation set has many molecules that are almost 85-90% similar to molecules in the train and test set. Can you highlight what could go wrong in the design of the task? As i said im also evaluating ROC-AUC, confusion matrix, accuracy, etc. All of which perform poorly. while training the model metric is also AUC, it is a binary classification and has applied regularization too. I can provide more details if needed. $\endgroup$
    – As13
    May 10, 2022 at 10:42
  • $\begingroup$ How can I detect data leakage? I did not perform any data processing in the beginning so what can be methods of detecting data leakage if any? $\endgroup$
    – As13
    May 10, 2022 at 10:56
  • $\begingroup$ @As13 it's difficult to say what can go wrong. data leakage is not easily detectable in general, because it depends completely on the data and how it was collected. My first idea would be to try to measure the similarity of the validation and test set vs the training set. One way to do this would be to use clustering on all the instances together and then observe whether the ones from the validation set always end up separated from the training set, this would mean that they don't follow the same distribution. $\endgroup$
    – Erwan
    May 10, 2022 at 13:57
  • $\begingroup$ I don't know the data at all but it's clear that the instances in this validation set were not taken randomly, so the main design question is whether it makes sense to have this different sample separate: would it make sense to redo a random selection of the data among all the instances? this way the model would be trained also on some instances that it currently doesn't know. This would probably work better, but it's important to make this decision only for good 'expert reasons', and I can't really help with this. $\endgroup$
    – Erwan
    May 10, 2022 at 14:03
  • $\begingroup$ Thanks Erwan for the detailed possibilities!! I will try to implement it. Thanks @Erwan $\endgroup$
    – As13
    May 11, 2022 at 4:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.