0
$\begingroup$

I am currently dealing with a classification problem for a massively imbalanced dataset. More specifically, it is a fraud detection dataset with around 290k rows of data, with distribution of 99.8% for class 0 (non-frauds) and 0.17% for class 1 (frauds).

I have been using XGBoost, Random Forest and LightBGM as my predictive models. I have also tried running the models differently by tuning class weights and resampling the dataset to bring it on a balanced scale. Moreover, I used f1-score, ROC-AUC score and a Precision-Recall curve as my main metrics, as it seems that other metrics are not representative of the result on an imbalanced dataset.

However, I seem to still be overfitting massively on my training data. In all scenarios, the f1-score, ROC-AUC score and the AP from Precision-Recall Curve of my training set are either 1.0 or 0.999, whereas those of the testing set are roughly around 0.85.

I wanted to ask whether this is a normal occurrence for an imbalanced dataset, and if not, is there any other method for me to fix it.

I would appreciate any response, and thank you all very much!

$\endgroup$

2 Answers 2

0
$\begingroup$

I am also facing the same issue in the Intrusion Detection System. I found the following suggestions that can be useful:

  • Reduce the number of datapoints with 0s to reduce the imbalance in dataset - Downsampling.
  • Obtain more data for the minor class (might not be possible, but is one solution)
  • Using Confusion matrix to analyze the model performance as well as the metrics you are already using
  • Using either Specificity or Recall as a metric to train the model
$\endgroup$
0
$\begingroup$

It's not uncommon to encounter overfitting when working with imbalanced datasets, but there are a few techniques you can try to mitigate this issue. Here are some suggestions from my personal experience:

-Hyperparameter tuning: Perform a systematic search for the best hyperparameters for your models, such as grid search or random search, to reduce overfitting. You can use cross-validation (e.g., k-fold or stratified k-fold) to ensure that the search is robust.

-Ensemble methods: Try using bagging or boosting techniques, which can help reduce overfitting by combining predictions from multiple base classifiers. In your case, you're already using ensemble models (XGBoost, RandomForest, and LightGBM). However, you can still try to create an ensemble of these models to see if it further improves performance.

-Feature selection: Identify and remove irrelevant or highly correlated features that might be causing your model to overfit. Techniques like Recursive Feature Elimination, LASSO regression, or correlation analysis can help you select the most important features.

-Early stopping: Implement early stopping in your training process to prevent the models from overfitting. This can be done using the built-in early stopping functionality in XGBoost and LightGBM. You'll need to set a validation dataset and specify a metric for evaluation (e.g., 'auc' or 'f1').

-Regularization: Regularization techniques like L1 or L2 regularization can help prevent overfitting by adding a penalty to the loss function. Both XGBoost and LightGBM support regularization parameters that can be adjusted during training.

-Adjust the decision threshold: Instead of using the default 0.5 threshold for classifying instances, you can experiment with different decision thresholds to improve the balance between precision and recall.

-Cost-sensitive learning: Assign different misclassification costs to the positive and negative classes during training. This can help your model learn to better predict the minority class. Most tree-based algorithms like XGBoost, RandomForest, and LightGBM support this approach.

-Try other resampling techniques: Experiment with different resampling techniques like SMOTE, ADASYN, or random oversampling of the minority class, and random undersampling of the majority class to balance the dataset. Be sure to perform these operations on the training set only and not the validation/test set.

Keep in mind that it may be difficult to achieve perfect performance on an imbalanced dataset, but the goal is to minimize the generalization error as much as possible. It's essential to monitor your model's performance on a separate validation set during training and select the model with the best performance on the validation set to avoid overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.