Dissmissing features based on correlation with target variable

Question

Is it valid to dismiss features based on their Pearson correlation values with the target variable in a classification problem?

say for instance I have a dataset with the following format where the target variable takes 1 or 0:

>>> dt.head()
   ID  var3  var15  imp_ent_var16_ult1  imp_op_var39_comer_ult1  \
0   1     2     23                   0                        0   
1   3     2     34                   0                        0   
2   4     2     23                   0                        0   
3   8     2     37                   0                      195   
4  10     2     39                   0                        0   

   imp_op_var39_comer_ult3  imp_op_var40_comer_ult1  TARGET  
0                        0                        0       0  
1                        0                        0       0  
2                        0                        0       0  
3                      195                        0       0  
4                        0                        0       0 

Computing the correlation matrix gives the following values

ID	var3	var15	imp_ent_var16_ult1	imp_op_var39_comer_ult1	imp_op_var39_comer_ult3	imp_op_var40_comer_ult1	TARGET
ID	1.0	-0.00102533166614	-0.00213549813966	-0.00311137548461	-0.00143645708778	-0.00413114484307	-0.00727672024906
var3	-0.00102533166614	1.0	-0.00445177129541	0.0018681447614	0.00598903116859	0.00681691701467	0.00151753041397
var15	-0.00213549813966	-0.00445177129541	1.0	0.0437222608106	0.0947624170998	0.101177078747	0.0427540973727
imp_ent_var16_ult1	-0.00311137548461	0.0018681447614	0.0437222608106	1.0	0.0412213212518	0.0348787079026	0.00989582043194
imp_op_var39_comer_ult1	-0.00143645708778	0.00598903116859	0.0947624170998	0.0412213212518	1.0	0.886476049204	0.342709191344
imp_op_var39_comer_ult3	-0.00413114484307	0.00681691701467	0.101177078747	0.0348787079026	0.886476049204	1.0	0.316671244555
imp_op_var40_comer_ult1	-0.00727672024906	0.00151753041397	0.0427540973727	0.00989582043194	0.342709191344	0.316671244555	1.0
TARGET	0.0031484687227	0.00447479817554	0.101322098561	-1.74602537678e-05	0.0103531295754	0.0035169224417	0.00311938694896

Is it valid, to dismiss all features where the correlation with target is lower than a threshold (say for instance, 0.1)?

What if there is a strong inter-attributes correlation as high as 1 where the correlated attributes are continuous variables, does this mean that these features hold redundant information for the learner? can I safely remove one of them without risking to lose information?

Answer 1

You've really got a classification problem on your hands, not a regression problem. Your target is not continuous, and Pearson correlation measures a relationship between continuous variables really. That's problematic enough to start.

Low correlation means there's no linear relationship; it doesn't mean there's no information in the feature that predicts the target.

I think you're really looking for mutual information, in this case between continuous and categorical variables. (I assume your other inputs are continuous?) This is a little involved; see https://stats.stackexchange.com/questions/29489/how-do-i-study-the-correlation-between-a-continuous-variable-and-a-categorical

If you're attempting to do feature selection then you could perform a logistic regression with L1 regularization and select features based on the absolute value of their coefficients.

Answer 2

Please note that Pearson correlation (and mutual information) considers the concept and the single feature.

There are cases in which a single feature is useless but given more features it becomes important.

Consider a concept which is the XOR of some features. Given all the features, the concept is totally predictable. Given one of them, you have 0 MI.

A more real life example is of age at death. Birth date and death date give you the age. One of them will have very low correlation (due to increase in life expectancy).