Choosing best predictors neural networks

there,

I'm currently working on a project where my database has about 120 patterns with 39 columns and I am trying to build a predictive neural network out of it. It's regression task.

I was trying to get the best predictors ( alone or combinations of'em) in a simple network( 3 neurons only) to then use cross validation to better tune the model. Problem are 1) the powerset of it is huge and my computer can't even handle generating the whole subset for simple fitting 2) only 3 neurons are already giving pretty poor results (r2<0)

Does somebody know a method or could please recommend a reading on choosing predictors for neural networks ?

Setup: windows 10, using MLPRegressor from scikitleran with hyperparameters Hidden layer sizes = 3, max_iter= 5000, solver='sgd'

A very quick way is to run some Tree-based ML model on your data, such as Random Forest or XGBoost. Tree-based models can return importance coefficients, estimating the relative explanatory power of each variable. You can implement a very large and deep ensemble of trees (we don't really care about overfitting at this point) so they return you the three strongest predictors. You can then take them and feed them into a Neural Network.

Another, more time consuming method is to run the model multiple timesa and substitute each variable, in alternation, with random noise with the same mean and standard deviation of the original variable. This perturbation method will tell you how much performance decreases when one variable is replaces by noise. This is accurate but very time consuming.

• Hi, @Leevo. I'm familiar the random forest feature importance, but does the output of such methods hold true for neural networks? As far as I knew from tree methods, it relies heavily on how deep you allow the tree to grow. I wasn't aware of the second method, I read about creating p Models excluding of one of the IV at time and check how the overall performance of the model changes. But didn't seem a solid approach – Lucas Abreu Sep 23 '19 at 10:54
• It can hold true for Neural Networks. Usually, if a variable is significantly more important than the others, it should result from any model you employ. I forgot to say that tree-base models suffer from multicollinearity problems as regressions do. So if your IVs show high collinearity, it might not be a good technique. The last model you mentioned could work, but it seems very time consuming. – Leevo Sep 23 '19 at 11:21
• Feature importance for tree based models doesn't necessarily correspond to importance for other models - all it tells you is that in the context of XGboost or Random Forest, those were the most important features for that specific model. Also, using random forests etc. for feature selection introduces data leakage. – Victor Ng Oct 23 '19 at 13:39
• I know and agree with what you said. That's a trade off between and computational costs and getting the job done – Leevo Oct 23 '19 at 13:47

To me this sounds like a classification problem? You are trying to classify patterns based on some of your 'variables' (39 of them?) If that is the case first of all $$R^2$$ really is not the right measure. Depending on Distributions of your classes you might want to look at measure such as Accuracy, $$\mathrm{AUROC}$$ or an $$F_1$$-score.

Having said that I personally have never had any nice experiences with neural networks implemented in SciKit learn, if you want to definitely use neural networks Id look into something like Keras, a fairly simple neural network library. As a general rule with a neural network you wouldnt need to actually create all the combinations of predictors, technically this job (given enough hidden layers) will be done by the network. For your task, as far as I can tell a simple MLP could do. A code example from tensorflow.keras import layers

model = tf.keras.Sequential()