3
$\begingroup$

I am working on a certain insurance claims related data-set to classify newly acquired customers as either claim or non-claim.

The basic problem with the training set is the extremely large imbalance in claim and non-claim profiles, with the claims amounting to just ~ 0.26% of the training set. Also, most claims are concentrated largely towards the final few years (data is sorted by date).

On applying Logistic Regression or even Random Forests, to train on 70% of the data, the test results were well below satisfactory.

I've been looking at alternate models and I came across this blog post. A particular line that got my attention is:

GBM is better than rf_t. In the paper, the best classifier for two-class data sets was avNNet_t, with 83.0% accuracy

Although, no real clarification was given as to why that was. Can someone help me open this "blackbox"? Which model really works (in the case described above) and why?

Improve this question
$\endgroup$
8
  • $\begingroup$ Given that you could get 99.74% accuracy with a model that just predicted "no claim" for your case, why do you think the blog post is applicable? I suggest look at datascience.stackexchange.com/questions/1107/… and datascience.stackexchange.com/questions/4944/… to see if they help $\endgroup$
    – Neil Slater
    Jun 10, 2016 at 11:58
  • $\begingroup$ @NeilSlater the purpose of my assignment is, in fact, to predict the claims. I was looking at boosting methods as a possible solution since (from what I understand) it would overcome the high bias problem. Perhaps my understanding is incorrect (is it?). In any case, I will go through the recommended links. $\endgroup$
    – bbq
    Jun 10, 2016 at 12:15
  • $\begingroup$ The accuracy metric, which is the one you picked up on in the blog, is the total frequency of true positives plus true negatives. I was merely pointing out that claims of 83% accuracy in someone else's test data mean very little to you. What metric are you actually getting a poor result on and want to improve? Some models are better at dealing with imbalanced training sets without making changes. I don't think you will resolve your problem purely by selecting a different model. However, it is worth a try in a black-box kind of way - just train different models, use a cv set, and pick best. $\endgroup$
    – Neil Slater
    Jun 10, 2016 at 12:36
  • 1
    $\begingroup$ I'm not sure I could answer that, even with more knowledge of your situation. However, maybe someone else could give insight if they knew a fair bit more about your data and the data from the blog. In a hand-wavy sense (i.e. not good enough for an answer): Neural networks improve on logistic regression where there are strong non-linear relationships between features and target. Tree boosting models, like xgboost, also deal OK with non-linear relationships, and xgboost seems to beat RF routinely in recent Kaggle competitions. $\endgroup$
    – Neil Slater
    Jun 10, 2016 at 12:58
  • 1
    $\begingroup$ Most models though will need some pre-processing of training data, or changes to objective to help with such a strong skew. Note you can improve true positive rate to 100% by predicting every customer as "claim" - I guess you are not really just using true positive rate as your goal. What exactly is the metric you are using, and what value do you currently get that is disappointing? This is key, because without a pre-agreed useful metric, you will be hard pushed to decide whether a different model is an improvement. $\endgroup$
    – Neil Slater
    Jun 10, 2016 at 13:01

1 Answer 1

Reset to default
2
$\begingroup$

I believe in your case, predicting claim is more important than no claim. As you said you have you got 70% Accuracy on the training data, most of the time you might be doing wrong predictions in claim case because of less records, comparatively, what I would suggest is to make the data set balance or select a random balanced data set (20% each of claim and non-clan) and train a model using previous techniques you have applied and test it on the remaining data. If possible use different error measures with respect to your business case such as giving weights to the outcomes. If the accuracy is not improved, you can implement GBM techniques on this data. Most of the times GBM makes better predictions because it increases the randomness (white noise) in residuals by decreasing the similarity among residuals. You can apply many different models on this data and check if the accuracy is improved, eventually we should be able to understand the model to explain someone why they should use this model. Moreover, if you use feature engineered data with different models, there is a high probability that you will do better than the previous models. However, this depends on your business understanding.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.