Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ One thing not mentioned here why redundant features are removed is interpretability. You cannot interpret the coefficients of a linear regression if you have correlated covariates. But the prediction can be good nevertheless. $\endgroup$– Ricardo Magalhães CruzJun 27, 2016 at 16:09
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Doing that is a very good idea. The problem is that doing that is very hard. Feature selection is a NP-complete problem. The practical meaning as that we don't know any fast algorithm that can select only the needed feature.
In the other direction, omitting features that don't have mutual information (MI) with the concept might cause you to throw the features you need most. There are cases in which a single feature is useless but given more features it becomes important.
Consider a concept which is the XOR of some features. Given all the features, the concept is totally predictable. Given one of them, you have 0 MI.
A more real life example is of age at death. Birth date and death date give you the age. One of them will have very low correlation (due to increase in life expectancy).
In practice, omitting features with low MI is OK. Many learning algorithms are using MI so they won't be able to use the omitted variables anyway. As for the selection itself, there are many algorithms, usually heuristics or approximation algorithm that are quite handy.
-
$\begingroup$ can you please give an example of when removing a redundant feature produces a classifier that has less performance power? $\endgroup$ Jun 29, 2016 at 3:08
-
$\begingroup$ If in redundant you mean exactly the same, than you can safely remove them. If they are different, consider a case of n features with high MI among them and low MI with the concept. If you'll remove them, you'll lose the MI with the concept. For example, let say there is a basic feature x and all the other are x with probability p and the concept with probability (1-p), independently. $\endgroup$– DaLJun 29, 2016 at 6:39
$\begingroup$
$\endgroup$
3
As with many things, it depends. The specifics of the relationship of your variables to the domain they describe will dictate, and even then relationships may not be intuitive. Seemingly disparate features can have a significant effect when combined in what is known as feature extraction.
Automated feature engineering techniques can help you decide what features are significant if you have the time and resources available, particularly when it comes to testing the impact of combined features. Additionally, some methods have the benefit of embedded feature selection, wherein the algorithm itself tends to diminish the effect of insignificant variables, e.g.: lasso regression, regularized decision trees, random forests, etc.
Here's a nice primer: http://machinelearningmastery.com/an-introduction-to-feature-selection/
-
$\begingroup$ what exactly do you mean when you say relationships may not be intuitive? Can you please give me an example? $\endgroup$ Jun 25, 2016 at 6:17
-
$\begingroup$ My comment about intuitiveness was with respect to a priori feature selection, i.e.: when not using automated or embedded techniques. In some domains an expert may make an intuitive guess about what features are relevant (e.g.: temperature and ice cream sales), whereas in other domains such intuition is not possible (e.g.: gene expression microarrays; sports telemetry data for specific physical activities). There is, of course, a wide variety of possible cases in between. $\endgroup$ Jun 25, 2016 at 7:10
-
$\begingroup$ But, to wrap it back around to your question, it is a good idea to reduce features in order to reduce training time and reduce the chance of overfitting and bias, but only when not using embedded feature selection techniques or after performing an automated sweep to verify significance. Another short article $\endgroup$ Jun 25, 2016 at 7:20
$\begingroup$
$\endgroup$
2
The fact that a feature is redundant in the presence of another one, or is not informative enough to describe the target variable, is not necessarily a sign of that feature not being useful.
Indeed, it may be the case that such feature may be extremely informative when combined with another one, in spite of not being very useful when considered in isolation.
Therefore, when applying feature selection methods, you should also consider combinations of features.
However, and as pointed out by another answer to this question, finding the best combination of features is a NP-complete problem. Therefore, applying feature selection to individual features may be a good approximation. However, I'd rather apply a greedy approach (see for instance https://studentnet.cs.manchester.ac.uk/pgt/COMP61011/goodProjects/Shardlow.pdf for more information about the topic.)
EDIT to answer OP's comment:
a) The table below displays an extreme example of a feature that by itself is very informative, but in combination with others is totally redundant (feature_2). This is a regression problem in which we are trying to build a model to predict the "output" variable from "feature_1" and "feature_2".
| feature_1 | feature_2 | output |
|-----------|-----------|--------|
| 1 | 1 | 0.1 |
| 2 | 2 | 0.2 |
| 3 | 3 | 0.3 |
| 4 | 4 | 0.4 |
| 5 | 5 | 0.5 |
| 6 | 6 | 0.6 |
b) The example below shows an extreme example of a feature that may not be very informative by itself, but that is very informative together with another one (feature_2).
| feature_1 | feature_2 | output |
|-----------|-----------|--------|
| 1 | 1 | 0.1 |
| 2 | 2 | 0.25 |
| 3 | 1 | 0.3 |
| 4 | 2 | 0.45 |
| 5 | 1 | 0.5 |
| 6 | 2 | 0.65 |
answered Jun 24, 2016 at 7:56
-
$\begingroup$ can you please give counter examples for both cases i stated? I cant seem to understand this intuitively. $\endgroup$ Jun 25, 2016 at 6:12
-
$\begingroup$ Edited my answer to include some simple examples. $\endgroup$ Jul 4, 2016 at 12:16