0
$\begingroup$

Some papers report the correlation between features when building a model and some don't. Is there a need to check the correlation between features and target feature? It won't be easy if the number of features is high.

$\endgroup$
  • 1
    $\begingroup$ Not if you are already doing feature selection, which is advisable in your scenario. $\endgroup$ – Emre Mar 13 '18 at 23:40
  • $\begingroup$ Isn't that only possible if you have well Understanding of the underlying dataset? Also if there is no correlation then how can we justify that we are having similar/related columns somehow? @Emre $\endgroup$ – Aditya Mar 14 '18 at 0:39
  • $\begingroup$ You don't need to understand the data especially well to do feature selection. Eliminating redundant columns is part of feature selection $\endgroup$ – Emre Mar 14 '18 at 1:13
5
$\begingroup$

Not really, no. Sort of. It depends on how complex your model/data is.

It's entirely possible to have a situation where a feature taken in isolation will not be correlated with the target variable, but multiple features considered together will. This is why univariate correlation is unreliable for feature selection. A trivial case that demonstrates this is a bivariate model performing a binary classification where the positive class is bounded by the right upper and left lower quadrants, and the negative class is bounded by the left upper and right lower quadranta (i.e. the "XOR" pattern):

XOR data

If the input features have the same sign (x>0 & y>0 or x<0 & y<0), it's the positive class, else it's the negative class. But either feature in isolation is completely useless and uncorrelated with the target.

Additionally, modern models like deep neural networks are effectively capable of "learning their own features", i.e. constructing extremely complex features by developing abstractions from the raw inputs. The "final" features learned by such a model will likely be correlated with the target, but the input features need not be.

For example, if you consider the imagenet task (classifying a photo as a member of one of 10,000 classes), I'd be very surprised to learn that there's any correlation between the values of specific pixels and any target class. That's just not how we interpret photos. The value of the pixel in position [25, 10] should not have any correlation with whether or not the picture is a photo of a dog. But, if we think of the entire network before the output layer as a feature engineering system (such that the "classifier" is just the output layer), then the "features" provided by the penultimate layer (the inputs to "The Classifier") probably have some correlation with the target.

TL;DR: If a feature is correlated with the target, it probably contains information about the target that will be useful for modeling. But that does not mean uncorrelated features are useless. Reporting correlation when it's there is a simple way to demonstrate that there's a signal in that variable. But lack of correlation doesn't necessarily mean you should throw those features away. In fact, correlation doesn't even mean you should necessarily use that feature either: you can have multiple features correlated with the target that are highly correlated with each other, in which case you would probably only want one or a handful of that group in your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.