I'm currently working on a fraud detection data set. I'm evaluating my training data with a 10-skfold roc auc and an estimator of default param LightGBM. But, the problem is every time I try to create a new column by calculating a ratio of 2 column, my CV always drop even though the new column is theoretically will highlight a difference between a fraud or not.
The other problem is when I find 2 good (increasing CV) separate new feature by feature engineering when evaluating them independently, my CV actually decrease when I combine both of the feature on the same evaluation.
Is there anything wrong with my way of feature engineering? Right now my step is: 1. Create a new column based on other column 2. Evaluate CV with 10-skfold default param 3. If the CV is increased (relative to original data) then it's good feature else no.
Any help is appreciated.