In multiple linear regression there is an F-test which can be used to evaluate whether or not a covariate has a meaningful impact on a model. This is typically done through either a forward selection or backwards selection algorithm. Does such a meaningful process exist for neural networks as well? The only reason I ask is because working with neural nets is an inherently stochastic process, so I do not know how I should try and get accurate bounds for the F-statistic.
$\begingroup$
$\endgroup$
Add a comment
|
$\begingroup$
$\endgroup$
2
I usually try to do the following process, I don't know whether it has name or not:
- Try to find features which an expert can say what the label is without hesitating.
- Plot correlation matrix for your sample to investigate whether each feature has correlation with others or not. I do this process whenever the input features are so much. Then try to reduce the correlated features or try to apply
PCA. Consider the point thatPCAalone does not care about the labels, so it may find principal components that your data would be so difficult to be separated but not necessarily.
-
$\begingroup$ This set of notes,cs231n.github.io/neural-networks-2, (and lecture 6 in the videos) describes something similar to what you were saying $\endgroup$ – user184074 Feb 3 '18 at 1:43
-
$\begingroup$ @user184074 actually I guess it is a common approach. $\endgroup$ – Media Feb 3 '18 at 6:37
$\begingroup$
$\endgroup$
3
In addition to what was suggested by @Media, you may consider adding a softmax layer to your model right above your input layer. This is a way of visualizing which features are strongly associated with each response. Of course as you say, an F-test will not really be appropriate. This is an active area of research as far as I know.
answered Jan 27 '18 at 15:59
-
$\begingroup$ Oh wow, I just googled "neural network feature selection" and immediately ran into a bunch of papers on the subject. I guess I will start with citeseerx.ist.psu.edu/viewdoc/… $\endgroup$ – user184074 Feb 2 '18 at 1:33
-
$\begingroup$ True. That paper came out in 1999, though, according to Google Scholar. There may be some more up-to-date info out there. It's far more popular, AFAIK, to use regularization (e.g. l1 or l2 or elasticnet) instead of eliminating a feature entirely. $\endgroup$ – StatsSorceress Feb 2 '18 at 13:46
-
$\begingroup$ @StatsSorceress thanks for your answer, would you please provide links for works that already has used this? $\endgroup$ – Media Feb 11 '18 at 19:34
