In multiple linear regression there is an F-test
which can be used to evaluate whether or not a covariate has a meaningful impact on a model. This is typically done through either a forward selection or backwards selection algorithm. Does such a meaningful process exist for neural networks as well? The only reason I ask is because working with neural nets is an inherently stochastic process, so I do not know how I should try and get accurate bounds for the F-statistic
.
I usually try to do the following process, I don't know whether it has name or not:
- Try to find features which an expert can say what the label is without hesitating.
- Plot correlation matrix for your sample to investigate whether each feature has correlation with others or not. I do this process whenever the input features are so much. Then try to reduce the correlated features or try to apply
PCA
. Consider the point thatPCA
alone does not care about the labels, so it may find principal components that your data would be so difficult to be separated but not necessarily.
-
$\begingroup$ This set of notes,cs231n.github.io/neural-networks-2, (and lecture 6 in the videos) describes something similar to what you were saying $\endgroup$ – user184074 Feb 3 '18 at 1:43
-
$\begingroup$ @user184074 actually I guess it is a common approach. $\endgroup$ – Media Feb 3 '18 at 6:37
In addition to what was suggested by @Media, you may consider adding a softmax layer to your model right above your input layer. This is a way of visualizing which features are strongly associated with each response. Of course as you say, an F-test will not really be appropriate. This is an active area of research as far as I know.
-
$\begingroup$ Oh wow, I just googled "neural network feature selection" and immediately ran into a bunch of papers on the subject. I guess I will start with citeseerx.ist.psu.edu/viewdoc/… $\endgroup$ – user184074 Feb 2 '18 at 1:33
-
$\begingroup$ True. That paper came out in 1999, though, according to Google Scholar. There may be some more up-to-date info out there. It's far more popular, AFAIK, to use regularization (e.g. l1 or l2 or elasticnet) instead of eliminating a feature entirely. $\endgroup$ – StatsSorceress Feb 2 '18 at 13:46
-
$\begingroup$ @StatsSorceress thanks for your answer, would you please provide links for works that already has used this? $\endgroup$ – Media Feb 11 '18 at 19:34