3
$\begingroup$

I used XGBoost to predict company's bankruptcy, which is an extremely unbalanced dataset. Although I tried weighting method as well as parameter tuning, the best result which I could obtain is as follows:

Best Parameters: {'clf__gamma': 0.1, 'clf__scale_pos_weight': 30.736842105263158, 'clf__min_child_weight': 1, 'clf__max_depth': 9}
Best Score: 0.219278428798
Accuracy: 0.966850828729
AUC: 0.850038850039
F1 Measure: 0.4
Cohen Kappa: 0.383129792673
Precision: 0.444444444444
recall: 0.363636363636

Confusion Matrix:
[[346   5]
 [  7   4]]

As the confusion matrix shows my model can not identify bankrupted companies very well, which results in poor performance measures such as precision, recall, Cohen's kappa, F measure. Also, I tried BlaggingClassifier, which is presented here. Really, it gives the following result:

Best Parameters: {'clf__n_estimators': 64}
Best Score: 0.133676613659
Accuracy: 0.809392265193
AUC: 0.819606319606
F1 Measure: 0.188235294118
Cohen Kappa: 0.142886555487
Precision: 0.108108108108
recall: 0.727272727273
Confusion Matrix: [[285  66]
 [  3   8]]

As it is shown, it predicts positive classes well but it does poor on negative class ( too many false positive). Could you please let me know how it is possible to combine these two classifier results in order to obtain a better result? A simple way to combine two classifier is to use a convex linear combination for predicted probabilities: t * p1 + (1 - t) * p2, where 0 <= t <= 1 and p1, p2 are predictions of the two classifiers. Then , I should search for an optimal value of t over a grid but I don't know how to do it?

I read that anomaly detection for example,one class svm and Isolation Forest could be used for extremely unbalanced data set, so could you please let me know how to do it i.e., by an example code? In general, I would appreciate if you could let me know how to deal with this issue.

Improve this question
$\endgroup$
2
  • $\begingroup$ I doubt that any method is going to perform well, given that it appears you have only about 10 negative examples in the entire training set. Your best hope is probably to obtain more data to train on. $\endgroup$
    – D.W.
    Dec 15, 2016 at 9:08
  • $\begingroup$ Thanks. I respectively have 25 and 11 positive examples in train set and test set. $\endgroup$
    – ebrahimi
    Dec 19, 2016 at 12:52

1 Answer 1

Reset to default
1
$\begingroup$

For starters, I am not confident that combining the results will give you what you expect. Have you checked if the true negative remain the same in both occurrences?

Moreover, have you tried to alternate the hyper parameters on the XGBoost model responsible for class balancing like max delta step or scale pos weight.

You can also try to use Sampling techniques for Over Sampling and Under Sampling. Or try possibly using many trees on XGBoost on a different random Under Sample.

P.S. I am interested on the anomaly detection paper you mentioned, could you provide a link, thnx.

Improve this answer
$\endgroup$
5
  • $\begingroup$ ).@Philip C.Thanks. Sampling techniques does not improve the result very much. Besides, I already tried to tune hyper-parameters of XGBoost. Anomaly detection paper is referred here: (svds.com/learning-imbalanced-classes $\endgroup$
    – ebrahimi
    Nov 14, 2016 at 16:08
  • $\begingroup$ @ebrahimi Many thanks for the link, I see there is also a scikit approach for the isolation forest, that you can try. Sorry to hear that. I hoped that in the worse case scenario, applying major Under Sampling steps could possibly degrade your performance but enhance your minority's class Recall measure. Have a look at the Gini coefficient on each of your attribute to the outcome variable. If all your scores are relatively too low, maybe it is difficult to establish a context after all. $\endgroup$
    – 20-roso
    Nov 14, 2016 at 18:28
  • $\begingroup$ @ebrahimi Your problem seems to be a tough one, for these cases they use Extreme Value Theorem, have a look in [youtube.com/watch?v=EACkiMRT0pc] (youtube). However, I am not knowledgeable in that area and might deviate from your Supervised Learning scope as well. $\endgroup$
    – 20-roso
    Nov 14, 2016 at 18:32
  • $\begingroup$ Thanks. Since one class svm or isolation forest are unsupervised I don't know how to determine scoring in order to grid search over the hyper-parameters. However, here (stats.stackexchange.com/questions/193887/…) an answer provided by Dan Levin may be useful but I don't know how to implement it. @Philip C. $\endgroup$
    – ebrahimi
    Nov 15, 2016 at 14:03
  • $\begingroup$ @ebrahimi There is a lot of literature, what you can do, which I proposed in my answer, you can use XGBoost on different Random Under Sampled samples. See that survey. Moreover, I do not know how Isolation Forest works, I assume if it provides a rank for each point, perhaps you can see if that is correlated with the output variable in some way. $\endgroup$
    – 20-roso
    Nov 15, 2016 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.