0
$\begingroup$

A quick example for this: we have many feature and two of them are policy count and premium_total (for all policies). We are predicting the expected claim amount with GBM or RF. Both policy_count and premium_total are important features by the model and they are highly correlated (0,8+).

My guess would be not to remove any feature, since both have different meaning/information, on the other hand as the policy count increases, the total premium will increase too. Removing the "less" important feature will not improve the model, nor make it worse.

What are your suggestions?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

When dealing with correlated variables, I always delete them if they are 0.95+ correlated. In your case, 80% is still high but not that extreme. You have mentioned that as the policy count increases, the total premium increases as well. However, is there a possibility that the total premium increases, without the policy count to increase? If yes, I would say keep it. After all you said that both of them are important. I assume that the running time does not matter that much, if it does, you can consider removing it for the model to run faster, but it will not make that much of a difference. Keep in mind that if the report tells you that it does not improve the model, nor make it worse, that means that this small change will affect only a very small number of new unseen features that you will give the model, so you are fine both ways.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! Yes, it is definitely possible that a single policy can have a large premium, while in other cases there are mulitple low premium policies. $\endgroup$
    – morqueatsz
    Commented Jan 19, 2023 at 10:22
  • $\begingroup$ In this case, I would keep both of them. Good luck with your project! $\endgroup$
    – Nemo_the_scientist
    Commented Jan 19, 2023 at 10:27
  • $\begingroup$ @Nemo_the_scientist You say I always delete them if they are 0.95+ correlated. In your case, 80% is still high but not that extreme, these are some interesting numbers you chose, why precisely these numbers? What's special about 95 % that you should remove variables? Why is 80% high but not extreme? Also, why are you using specifically (pearson) correlation? Why not mutual information or something else? This is quite arbitrary. $\endgroup$
    – user2974951
    Commented Jan 19, 2023 at 10:57
  • $\begingroup$ @user2974951 Hi. Its from experience. There is nothing specific about 95%. I have noticed that removing a feature with 0.95+ correlation always results in higher accuracy. For your next question, 80% correlation is considered a fairly strong linear relationship in statistics. The best way to understand it, is to look at plots. You can clearly see the linear relationship at 80%. I use Pearson every time the data is normally distributed and does not have a lot of outliers. If it is not normally distributed or has lots of outliers I use Spearman. I do not have experience with other methods. $\endgroup$
    – Nemo_the_scientist
    Commented Jan 19, 2023 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.