Choosing best predictors neural networks

Question

there,

I'm currently working on a project where my database has about 120 patterns with 39 columns and I am trying to build a predictive neural network out of it. It's regression task.

I was trying to get the best predictors ( alone or combinations of'em) in a simple network( 3 neurons only) to then use cross validation to better tune the model. Problem are 1) the powerset of it is huge and my computer can't even handle generating the whole subset for simple fitting 2) only 3 neurons are already giving pretty poor results (r2<0)

Does somebody know a method or could please recommend a reading on choosing predictors for neural networks ?

Setup: windows 10, using MLPRegressor from scikitleran with hyperparameters Hidden layer sizes = 3, max_iter= 5000, solver='sgd'

Answer 1

A very quick way is to run some Tree-based ML model on your data, such as Random Forest or XGBoost. Tree-based models can return importance coefficients, estimating the relative explanatory power of each variable. You can implement a very large and deep ensemble of trees (we don't really care about overfitting at this point) so they return you the three strongest predictors. You can then take them and feed them into a Neural Network.

Another, more time consuming method is to run the model multiple timesa and substitute each variable, in alternation, with random noise with the same mean and standard deviation of the original variable. This perturbation method will tell you how much performance decreases when one variable is replaces by noise. This is accurate but very time consuming.

Answer 2

To me this sounds like a classification problem? You are trying to classify patterns based on some of your 'variables' (39 of them?) If that is the case first of all $R^2$ really is not the right measure. Depending on Distributions of your classes you might want to look at measure such as Accuracy, $\mathrm{AUROC}$ or an $F_1$-score.

Having said that I personally have never had any nice experiences with neural networks implemented in SciKit learn, if you want to definitely use neural networks Id look into something like Keras, a fairly simple neural network library. As a general rule with a neural network you wouldnt need to actually create all the combinations of predictors, technically this job (given enough hidden layers) will be done by the network. For your task, as far as I can tell a simple MLP could do. A code example from tensorflow.keras import layers

model = tf.keras.Sequential()
model.add(layers.Dense(64, activation='relu'))
model.add(layer.Dropout(0.5))
model.add(layers.Dense(64, activation='relu'))
model.add(layer.Dropout(0.5))
model.add(layers.Dense(10, activation='softmax'))
model.compile(optimizer=tf.keras.optimizers.Adam(0.01),
          loss='categorical_crossentropy',
          metrics=['accuracy'])

However, from what I see this seems to be fairly structured data, that could just as well be analyzed with something like Gradient Boosted Trees or else. You might want to look at those as well, could give better results.