I'm trying to do a correlation analysis between inputs and outputs inspecting the data in order to understand which input variables to include. What could be a threshold in the correlation value to consider a variable eligible to be an input for my Neural Network?
Given non-linearity of neural networks, I believe correlation analysis isn't a good way to estimate importance of variables. For example, imagine that you have 2 input variables -
x2 - and following conditions hold:
cor(x2, y) = 1if
x1 = 1
cor(x2, y) = 0otherwise
x1 = 1in 10% of cases
x2 is a very good predictor for
y, but only given that
x1 = 1, which is the case only in 10% of data. Taking into account correlations of
x2 separately won't expose this dependency, and you will most likely drop out both variables.
There are other ways to perform feature selection, however. Simplest one is to train your model with all possible sets of variables and check the best subset. This is pretty inefficient with many variables, though, so many ways to improve it exist. For a good introduction in best subset selection see chapter 6.1 of Introduction to Statistical Learning.