3
$\begingroup$

I have seen researchers using pearson's correlation coefficient to find out the relevant features -- to keep the features that have a high correlation value with the target. The physical implication according to my understanding is that the correlated features contribute more information in finding out the target in regression and classification problems. Whereas, we remove the features which are redundant and have very negligible correlation value. Please correct me if wrong. However, the calculation of correlation coefficient is based on the assumption that the dataset is linear.

Q1) Should highly correlated features with the target variable be included or removed from classification and regression problems? Is there a better/elegant explanation to this step?

Q2) How do we know that the dataset is linear when there are multiple variable involved? What does it mean by dataset being linear?

Q3) How to do feature importance for nonlinear case?

$\endgroup$
4
$\begingroup$

Q1) Should highly correlated features with the target variable be included or removed from classification and regression problems? Is there a better/elegant explanation to this step?

Actually there's no strong reason either to keep or remove features which have a low correlation with the target response, other than reducing the number of features if necessary:

  • It is correct that correlation is often used for feature selection. Feature selection is used for dimensionality reduction purposes, i.e. mostly to avoid overfitting due to having too many features / not enough instances (it's a bit more complex than this but that's the main idea). My point is that there's little to no reason to remove features if the number of features is not a problem, but if it is a problem then it makes sense to keep only the most informative features, and high correlation is an indicator of "informativeness" (information gain is another common measure to select features).
  • In general feature selection methods based on measuring the contribution of individual features are used because they are very simple and don't require complex computations. However they are rarely optimal because they don't take into account the complementarity of groups of features together, something that most supervised algorithms can use very well. There are more advanced methods available which can take this into account: the most simple one is a brute-force method which consists in repeatedly measuring the performance (usually with cross-validation) with any possible subset of features... But that can take a lot of time for a large set of features.

However features which are highly correlated together (i.e. between features, not with the target response), should usually be removed because they are redundant and some algorithms don't deal very well with those. It's rarely done systematically though, because again this involves a lot of calculations.

Q2) How do we know that the dataset is linear when there are multiple variable involved? What does it mean by dataset being linear?

It's true that correlation measures are based on linearity assumptions, but that's rarely the main issue: as mentioned above it's used as an easy indicator of "amount of information" and it's known to be imperfect anyway, so the linearity assumption is not so crucial here.

A dataset would be linear if the response variable can be expressed as a linear equation of the features (i.e. in theory one would obtain near-perfect performance with a linear regression).

Q3) How to do feature importance for nonlinear case?

Information gain, KL divergence, and probably a few other measures. But using these to select features individually is also imperfect.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer, it really helped. Two last questions as a follow up -- (1) is feature reduction done after standardization or normalization or on the raw data set?(2) Often based on practice I noticed that for regression problem, if the target response is transformed to logarithm base 10, the fit is better. Why is that? $\endgroup$ – Srishti M Nov 22 at 1:47
  • 1
    $\begingroup$ To be honest I'm not really sure about these two questions: (1) my intuition would be to standardize first, because this way the features selection takes into account the features exactly as they would be used. However I suspect that it doesn't matter too much, since standardization shouldn't change the correlation with the response variable too much. (2) I'm not sure if this is really common, it probably depends on the task, but I assume that the reason would be to convert from a non-linear relation to a (more) linear one: many problems are not linear but can be transformed into a linear one. $\endgroup$ – Erwan Nov 22 at 1:59
2
$\begingroup$

for feature engineering there are different methods.

Pearson Correlation comes under Filter methods. Filter methods gives intuition on the high level. This can be the first step for feature engineering. In this process

  • the features having high correlation with target should be considered.

  • the features having high correlation among themselves should also be removed as, "they are acting two independent variables doing same work" then why keep both.

After considering the correlation approaches you can also dig in to the Wrapper based methods which are more robust for feature selection but that includes the burden of training process.

Refer this for introduction to the different approaches.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.