1
$\begingroup$

I'm currently working on a fraud detection data set. I'm evaluating my training data with a 10-skfold roc auc and an estimator of default param LightGBM. But, the problem is every time I try to create a new column by calculating a ratio of 2 column, my CV always drop even though the new column is theoretically will highlight a difference between a fraud or not.

The other problem is when I find 2 good (increasing CV) separate new feature by feature engineering when evaluating them independently, my CV actually decrease when I combine both of the feature on the same evaluation.

Is there anything wrong with my way of feature engineering? Right now my step is: 1. Create a new column based on other column 2. Evaluate CV with 10-skfold default param 3. If the CV is increased (relative to original data) then it's good feature else no.

Any help is appreciated.

$\endgroup$
1
$\begingroup$

You might be dealing with overfitting, since your model does not generalize that well. It should be noted that Light GBM is sensitive to overfitting in small datasets.

To solve this, you can check for features with correlation and drop some of them. Another solution can be the implementation of a feature selection method. Lastly, since GBM is a tree algorithm, you can try an early termination.

If you are dealing with highly imbalance data, you can consider using a stratified method rather than CV. In sklearn, there are StratifiedKFold and StratifiedShuffleSplit, which ensure that relative class frequencies are approximately preserved in both train and validation phases.

| improve this answer | |
$\endgroup$
0
$\begingroup$

I think I have enough experience on doing feature engineering on the data for the training of the models in LightGBM. When feature engineering, it is wise to logically reason about the features that you create and how they should effect your model, just like you do; you logically expect that the ratio will work. However, the results may not be as you expected, as you experience now.

1) Always visualize your data after processing; divisions by zero can break your model, or create harmful outliers by incredibly small divisors instead of zero. Therefore, ratios can be dangerous as features Alternative ideas can variate around the difference between two, square difference, their sum, their square sum or their multiplication.

I'd also recommend you to visualize statistical properties of the features by plotting the distribution or using a tool like pandas.Describe(). Then you will be able to see whether there are outliers. ( for example, 90% of the features are distributed in the range 10 to 170, where some are higher than 10,000 or so)

2) A new feature may harm your model by its own, but new patterns can occur with several new features combined together that your ML model can explicitly see. First create the features in your mind, then try their combinations.

3) Always print your training and test multi_error during the iterations, verbose_eval = 10 is a viable parameter that you can give to the train() function of LightGBM; you will see the training and test errors at every 10 iterations upon training. By doing that, you will be able to observe the changes on the performance by the features without even finishing up the iterations, and whether your model overfits.

Hope I could help.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.