0
$\begingroup$

I am currently working on a very imbalanced dataset:

  • 24 million transactions (rows of data)
  • 30,000 fraudulent transactions (0.1% of total transactions)

and I am using XGBoost as the model to predict whether a transaction is fraudulent or not. After tuning some hyperparameters via optuna, I have received such results

  • F1 Score on Training Data : 0.5881226277372263

  • F1 Score on Validation Data : 0.8699220352892901

  • ROC AUC score on Training Data : 0.9991431591607794

  • ROC AUC score on Validation Data : 0.9254554224474641

Although the ROC AUC score are quite high, the F1 score of my trainig data is quite low and its ROC AUC score is abnormally high. I was wondering what is wrong with my model, or my data? Am I overfitting, and are these results acceptable?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ how did you split your training/validation data? looking at f1 scores it seems that the model is underfitting, while the AUC tells us that it is overfitting. in general, for a model that would generalise well, the train and validation errors should be as close as possible. given the extreme imbalance, i suggest (if you haven't already) to split the data with stratification $\endgroup$
    – Stefan Popov
    Commented May 22, 2023 at 10:39
  • 1
    $\begingroup$ Can you please provide the actual ROC curves, and the precision and recall separately? $\endgroup$
    – Ben Reiniger ♦
    Commented May 22, 2023 at 13:35
  • 1
    $\begingroup$ @StefanPopov I split the data based on the feature Year, the data was indexed by time. I have around 20 million for training, around 1.5 for validation and testing. I have tried random splitting before, with decent results, but I believe it would not be able to generalize well with future data. $\endgroup$
    – Hai Nguyen
    Commented May 23, 2023 at 20:05
  • $\begingroup$ @BenReiniger I will update them shortly, thank you! $\endgroup$
    – Hai Nguyen
    Commented May 23, 2023 at 20:12

1 Answer 1

Reset to default
2
$\begingroup$

Your model is overfitting. This is evident from the high ROC AUC score on the training data and the lower F1 score on the validation data. The ROC AUC score is a measure of the model's ability to distinguish between the positive and negative classes, while the F1 score is a measure of the model's precision and recall. The fact that the ROC AUC score is so high on the training data indicates that the model is learning to identify the positive class very well, but it is also learning to identify the negative class very well. This is a sign of overfitting, as the model is learning to identify patterns in the training data that are not generalizable to the validation data.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Yes @M__, thanks for the correction. $\endgroup$
    – Bayrem
    Commented May 22, 2023 at 13:08
  • $\begingroup$ I had thought that to overfit, the training metrics would be high while those of the testing or validation would be low, or does overfitting just mean that the model can not identify future data as well as the training one? I have been trying to cross validate and do hyperparameters tuning, with much better results, however the ROC AUC and even the PR AUC of the training data remain extremely high, around 0.99x, does this necessarily indicate overfitting, and what should I do next? $\endgroup$
    – Hai Nguyen
    Commented May 23, 2023 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.