5
$\begingroup$

Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem?

say for instance I have a dataset with the following format where the target variable takes 1 or 0:

>>> dt.head()
   ID  var3  var15  imp_ent_var16_ult1  imp_op_var39_comer_ult1  \
0   1     2     23                   0                        0   
1   3     2     34                   0                        0   
2   4     2     23                   0                        0   
3   8     2     37                   0                      195   
4  10     2     39                   0                        0   

   imp_op_var39_comer_ult3  imp_op_var40_comer_ult1  TARGET  
0                        0                        0       0  
1                        0                        0       0  
2                        0                        0       0  
3                      195                        0       0  
4                        0                        0       0 

Computing the correlation matrix gives the following values

|ID|var3|var15|imp_ent_var16_ult1|imp_op_var39_comer_ult1|imp_op_var39_comer_ult3|imp_op_var40_comer_ult1|TARGET
ID|1.0|-0.00102533166614|-0.00213549813966|-0.00311137548461|-0.00143645708778|-0.00413114484307|-0.00727672024906|0.0031484687227
var3|-0.00102533166614|1.0|-0.00445177129541|0.0018681447614|0.00598903116859|0.00681691701467|0.00151753041397|0.00447479817554
var15|-0.00213549813966|-0.00445177129541|1.0|0.0437222608106|0.0947624170998|0.101177078747|0.0427540973727|0.101322098561
imp_ent_var16_ult1|-0.00311137548461|0.0018681447614|0.0437222608106|1.0|0.0412213212518|0.0348787079026|0.00989582043194|-1.74602537678e-05
imp_op_var39_comer_ult1|-0.00143645708778|0.00598903116859|0.0947624170998|0.0412213212518|1.0|0.886476049204|0.342709191344|0.0103531295754
imp_op_var39_comer_ult3|-0.00413114484307|0.00681691701467|0.101177078747|0.0348787079026|0.886476049204|1.0|0.316671244555|0.0035169224417
imp_op_var40_comer_ult1|-0.00727672024906|0.00151753041397|0.0427540973727|0.00989582043194|0.342709191344|0.316671244555|1.0|0.00311938694896
TARGET|0.0031484687227|0.00447479817554|0.101322098561|-1.74602537678e-05|0.0103531295754|0.0035169224417|0.00311938694896|1.0

Is it valid, to dismiss all features where the correlation with target is lower than a threshold (say for instance, 0.1)?

What if there is a strong inter-attributes correlation as high as 1 where the correlated attributes are continuous variables, does this mean that these features hold redundant information for the learner? can I safely remove one of them without risking to lose information?

$\endgroup$
10
$\begingroup$

You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.

Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts the target.

I think you're really looking for mutual information, in this case between continuous and categorical variables. (I assume your other inputs are continuous?) This is a little involved; see https://stats.stackexchange.com/questions/29489/how-do-i-study-the-correlation-between-a-continuous-variable-and-a-categorical

If you're attempting to do feature selection then you could perform a logistic regression with L1 regularization and select features based on the absolute value of their coefficients.

$\endgroup$
  • $\begingroup$ Thanks for answering! What if there is a strong inter-attributes correlation as high as 1, does this mean that these features hold redundant information for the learner? can I safely remove one of them without risking to lose information? $\endgroup$ – MedAli Mar 12 '16 at 15:58
2
$\begingroup$

Please note that Pearson correlation (and mutual information) considers the concept and the single feature.

There are cases in which a single feature is useless but given more features it becomes important.

Consider a concept which is the XOR of some features. Given all the features, the concept is totally predictable. Given one of them, you have 0 MI.

A more real life example is of age at death. Birth date and death date give you the age. One of them will have very low correlation (due to increase in life expectancy).

$\endgroup$
  • $\begingroup$ Thanks for answering! In such case, is there a systematic way to identify when a combination of features have more "predictive power" when put together? there are few tips here, blog.datadive.net/… which other methods are there to accomplish this? $\endgroup$ – MedAli Mar 13 '16 at 18:41
  • $\begingroup$ What you are asking for is en.wikipedia.org/wiki/Feature_selection . Solving it accurately is NP-Complete. however, there are plenty useful methods that can help in practice. $\endgroup$ – DaL Mar 14 '16 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.