3
$\begingroup$

In multiple linear regression there is an F-test which can be used to evaluate whether or not a covariate has a meaningful impact on a model. This is typically done through either a forward selection or backwards selection algorithm. Does such a meaningful process exist for neural networks as well? The only reason I ask is because working with neural nets is an inherently stochastic process, so I do not know how I should try and get accurate bounds for the F-statistic.

$\endgroup$
2
$\begingroup$

I usually try to do the following process, I don't know whether it has name or not:

  • Try to find features which an expert can say what the label is without hesitating.
  • Plot correlation matrix for your sample to investigate whether each feature has correlation with others or not. I do this process whenever the input features are so much. Then try to reduce the correlated features or try to apply PCA. Consider the point that PCA alone does not care about the labels, so it may find principal components that your data would be so difficult to be separated but not necessarily.
$\endgroup$
2
  • $\begingroup$ This set of notes,cs231n.github.io/neural-networks-2, (and lecture 6 in the videos) describes something similar to what you were saying $\endgroup$ – user184074 Feb 3 '18 at 1:43
  • $\begingroup$ @user184074 actually I guess it is a common approach. $\endgroup$ – Media Feb 3 '18 at 6:37
1
$\begingroup$

In addition to what was suggested by @Media, you may consider adding a softmax layer to your model right above your input layer. This is a way of visualizing which features are strongly associated with each response. Of course as you say, an F-test will not really be appropriate. This is an active area of research as far as I know.

$\endgroup$
3
  • $\begingroup$ Oh wow, I just googled "neural network feature selection" and immediately ran into a bunch of papers on the subject. I guess I will start with citeseerx.ist.psu.edu/viewdoc/… $\endgroup$ – user184074 Feb 2 '18 at 1:33
  • $\begingroup$ True. That paper came out in 1999, though, according to Google Scholar. There may be some more up-to-date info out there. It's far more popular, AFAIK, to use regularization (e.g. l1 or l2 or elasticnet) instead of eliminating a feature entirely. $\endgroup$ – StatsSorceress Feb 2 '18 at 13:46
  • $\begingroup$ @StatsSorceress thanks for your answer, would you please provide links for works that already has used this? $\endgroup$ – Media Feb 11 '18 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.