I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to test the network with the new feature and see the results, but is there a method to test if the feature IS UNLIKELY helpful? Like correlation measures etc?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ A non-random correlation might be an indicator that the feature is useful. But I'm not so sure about pre-training tests that could rule ideas out. The paper you link makes it clear that non-linear correlations are not well detected by the available tests, but a neural net has a chance of finding and using them. $\endgroup$– Neil SlaterJul 10, 2014 at 11:28
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information. A low correlation between the new feature and existing features is likely preferable.
A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be valuable, but the absence of a high correlation is not necessary a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.
If the new feature was manually constructed from a combination of existing features, consider leaving it out. The beauty of neural networks is that little feature engineering and preprocessing is required -- features are instead learned by intermediate layers. Whenever possible, prefer learning features to engineering them.
answered Jul 10, 2014 at 15:43
-
$\begingroup$ I always thought to compare the value to predict with the features, you are talking about correlation between features. Is your answer applicable also to my case? in theory I should add only new features that are correlated to the value to predict, right? $\endgroup$ Jul 10, 2014 at 19:06
-
$\begingroup$ That's also a valuable metric -- just updated my answer to address that as well. $\endgroup$ Jul 10, 2014 at 19:18
-
$\begingroup$ In short, strong correlations with the value to predict is a great sign, but weak correlation with the value to predict is not necessarily a bad sign. $\endgroup$ Jul 10, 2014 at 19:19
-
$\begingroup$ Thanks. I'm writing a report and I wanted to show the linear/non-linear correlations in order to justify the features (even before the results). Does it make any sense? From your answer I could make a matrix of correlations but maybe it's nosense $\endgroup$ Jul 10, 2014 at 19:27
-
1$\begingroup$ I would use non-linear correlations, but ok thanks $\endgroup$ Jul 10, 2014 at 23:17
