I am working on a classification problem, where I am trying to predict a fraud login. The data is highly imbalanced i.e.
0 = non fraud logins , 1 = fraud logins

0 : 4538076

1 : 365

I have been trying to model an XGBoost on this data . I have around 30 features. One such feature has the distribution as follows : (Most of the features have a distribution like this where we can clearly see the numbers are higher for frauds but there is also a considerable amount of overlap among two groups. I cannot put all features here so just an example of one feature)

enter image description here

Also attaching a KDE plot for another feature to give an idea about the data. enter image description here

So,first I tried a borderline SMOTE, and undersampled few from the majority class too , the final distribution for train data is as follows: After OverSampling, counts of label '1': 4877 After OverSampling, counts of label '0': 97540

With this , I have train, test and validation data(test and validation data are not resampled) and trained xgboost with the following parameters with a RandomizedCV.

# A parameter grid for XGBoost
params = {
        'min_child_weight': [1,5,10],
        'gamma': [0.5,0.4,1,2,2.5,1.5],
        'subsample': [0.8,0.95, 1.0],
        'colsample_bytree': [0.6,0.7 ,0.8,0.9,1.0],
        'max_depth': [3, 4, 5],

xgb = XGBClassifier(objective='binary:logistic', nthread=-1 , eval_metric = 'logloss')
folds = 10
param_comb = 150

skf = StratifiedKFold(n_splits=folds, shuffle = True, random_state = 1001)

random_search = RandomizedSearchCV(xgb, param_distributions=params, n_iter=param_comb,scoring='f1_weighted', n_jobs=-1, cv = skf.split(X_train,y_train), verbose=3, random_state=1001 )

random_search.fit(X_train, y_train ,early_stopping_rounds = 10, eval_metric = 'logloss' , eval_set = [(df_val,y_val)])

Results :

Best estimator:
XGBClassifier(alpha=0.05, colsample_bylevel=0.6, colsample_bytree=0.8,
              eta=0.007, eval_metric='logloss', gamma=0.5, lambda=0.02,
              max_delta_step=6, max_depth=5, n_estimators=500, nthread=-1,

 Best normalized gini score for 10-fold search with 150 parameter combinations:

 Best hyperparameters:
{'subsample': 0.8, 'scale_pos_weight': 1, 'n_estimators': 500, 'min_child_weight': 1, 'max_depth': 5, 'max_delta_step': 6, 'lambda': 0.02, 'gamma': 0.5, 'eta': 0.007, 'colsample_bytree': 0.8, 'colsample_bylevel': 0.6, 'alpha': 0.05}

Results on train data :

enter image description here

Probability distribution on train data : enter image description here (I have even tried running a simple logistic regression with the labels and probabilities to calibrate it, doesn't work)

Results on test data :

# confusion matrix
matrix = confusion_matrix(y_test,preds_test , labels=[1,0])
print('Confusion matrix : \n',matrix)

# outcome values order in sklearn
tp, fn, fp, tn = confusion_matrix(y_test,preds_test,labels=[1,0]).reshape(-1)
print('Outcome values : \n', tp, fn, fp, tn)

# classification report for precision, recall f1-score and accuracy
matrix = classification_report(y_test,preds_test , labels=[1,0])
print('Classification report : \n',matrix)

enter image description here

What am I doing wrong? How do I improve the model? P.S. i am also exploring Isolation Forest / LOF but I dont think they aren't working out well in my case either. Sorry, about the long question but I needed to make sure I give enough context.

  • $\begingroup$ By improve, what metric(s) are you targeting? What is the performance (metrics) needed for the business problem? $\endgroup$ – Craig May 10 at 12:01
  • $\begingroup$ Main motive is to identify the frauds so increasing the true positives is important. But also false positives cannot be very high. Metrics I have been looking at are Precision, Recall, and PR-AUC. $\endgroup$ – Aditi May 10 at 15:04
  • $\begingroup$ SMOTE - I believe often trades off Precision for Recall. No free lunch. And SMOTE can connect inliers to outliers and clusters. Perhaps whatever cutoff value being used is not optimal. Those are non-proper scoring rules and need the cutoff optimized until those metrics are meaningful. Are there other ways to measure this business problem? This is looking off how much fraud, but is there a how much fraud $ instead of quantity, or fraud rings vs individual fraud or fraud from new customers vs fraud from established customers vs... $\endgroup$ – Craig May 10 at 20:24
  • $\begingroup$ The probabilities are so low and so much skewed that a slight change in the cut offs drastically increases false positives. But probably a business rule can be overlayed to tackle that. Your second suggestion though is somethings useful, I have to think over it. This problem at hand is classifying each login attempt, but the login attempt is practically an account takeover attempt, so user metrics aren't much helpful here instead probably device metrics are more important(?). $\endgroup$ – Aditi May 11 at 12:07
  • $\begingroup$ You also have the standard feature engineering, getting more data/features, samples, etc. And maybe this is the best model that can be built given the data and target. I also work in fraud with very rare events and sometimes the best is the best and that is all. We build rules on top of the model, like you said, to handle specific cases where the model is weak. Depends on the FP vs FN tolerance. $\endgroup$ – Craig May 12 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.