Feature selection method explanation

Question

In the context of feature relevance, I am trying to understand the meaning of the correlation method for feature selection. Can somebody please explain if the following results of the correlation coefficients arise, then should I take that feature? The rule is to select the features for which corrcoeff values are greater than 0.5. Please correct me if wrong. The way I am calculating is using Matlab's corrcoeff(target,feature) where target and feature are vectors

Case1: corrcoeff returns NaN values --

Nan Nan
Nan  1

Should the feature be selected since the value is greater than 0.5?

Case2: corrcoeff returns 0 values

0 0
0  1

In this case, I should reject the feature.

Case3:

-0.3 0
0    -0.3

Negatively correlated but absolute values less than 0.5, so reject the feature

Case4: What if there is no linear relationship at all in which case corrcoeff will not work. How do I know if there is no linear relationship and in that case how to do feature selection; is there any other function or technique?

Answer 1

The idea is quite simple: If you find features that are correlated with your target variable but uncorrelated with each other, it will give you information with which you can predict your target variable.

The threshold of 0.5 is chosen by a person and should be changed depending on your data. For example, if you find 10 features that are uncorrelated with each other but all have a 0.3 absolute correlation coefficient with the S&P 500 than you found the stock market equivalent of the Holy Grail.

Also, as you already mentioned, the correlation coefficient only captures linear correlations. This could cause you to exclude strongly related features just because they have a more complex relationship to your target variable.

Therefore, I would strongly advise against this method. Instead, I recommend plotting the data in pairplots.