1
$\begingroup$

there,

I'm currently working on a project where my database has about 120 patterns with 39 columns and I am trying to build a predictive neural network out of it. It's regression task.

I was trying to get the best predictors ( alone or combinations of'em) in a simple network( 3 neurons only) to then use cross validation to better tune the model. Problem are 1) the powerset of it is huge and my computer can't even handle generating the whole subset for simple fitting 2) only 3 neurons are already giving pretty poor results (r2<0)

Does somebody know a method or could please recommend a reading on choosing predictors for neural networks ?

Setup: windows 10, using MLPRegressor from scikitleran with hyperparameters Hidden layer sizes = 3, max_iter= 5000, solver='sgd'

| improve this question | |
$\endgroup$
1
$\begingroup$

A very quick way is to run some Tree-based ML model on your data, such as Random Forest or XGBoost. Tree-based models can return importance coefficients, estimating the relative explanatory power of each variable. You can implement a very large and deep ensemble of trees (we don't really care about overfitting at this point) so they return you the three strongest predictors. You can then take them and feed them into a Neural Network.

Another, more time consuming method is to run the model multiple timesa and substitute each variable, in alternation, with random noise with the same mean and standard deviation of the original variable. This perturbation method will tell you how much performance decreases when one variable is replaces by noise. This is accurate but very time consuming.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, @Leevo. I'm familiar the random forest feature importance, but does the output of such methods hold true for neural networks? As far as I knew from tree methods, it relies heavily on how deep you allow the tree to grow. I wasn't aware of the second method, I read about creating p Models excluding of one of the IV at time and check how the overall performance of the model changes. But didn't seem a solid approach $\endgroup$ – Lucas Abreu Sep 23 '19 at 10:54
  • $\begingroup$ It can hold true for Neural Networks. Usually, if a variable is significantly more important than the others, it should result from any model you employ. I forgot to say that tree-base models suffer from multicollinearity problems as regressions do. So if your IVs show high collinearity, it might not be a good technique. The last model you mentioned could work, but it seems very time consuming. $\endgroup$ – Leevo Sep 23 '19 at 11:21
  • $\begingroup$ Feature importance for tree based models doesn't necessarily correspond to importance for other models - all it tells you is that in the context of XGboost or Random Forest, those were the most important features for that specific model. Also, using random forests etc. for feature selection introduces data leakage. $\endgroup$ – Victor Ng Oct 23 '19 at 13:39
  • $\begingroup$ I know and agree with what you said. That's a trade off between and computational costs and getting the job done $\endgroup$ – Leevo Oct 23 '19 at 13:47
0
$\begingroup$

To me this sounds like a classification problem? You are trying to classify patterns based on some of your 'variables' (39 of them?) If that is the case first of all $R^2$ really is not the right measure. Depending on Distributions of your classes you might want to look at measure such as Accuracy, $\mathrm{AUROC}$ or an $F_1$-score.

Having said that I personally have never had any nice experiences with neural networks implemented in SciKit learn, if you want to definitely use neural networks Id look into something like Keras, a fairly simple neural network library. As a general rule with a neural network you wouldnt need to actually create all the combinations of predictors, technically this job (given enough hidden layers) will be done by the network. For your task, as far as I can tell a simple MLP could do. A code example from tensorflow.keras import layers

model = tf.keras.Sequential()
model.add(layers.Dense(64, activation='relu'))
model.add(layer.Dropout(0.5))
model.add(layers.Dense(64, activation='relu'))
model.add(layer.Dropout(0.5))
model.add(layers.Dense(10, activation='softmax'))
model.compile(optimizer=tf.keras.optimizers.Adam(0.01),
          loss='categorical_crossentropy',
          metrics=['accuracy'])

However, from what I see this seems to be fairly structured data, that could just as well be analyzed with something like Gradient Boosted Trees or else. You might want to look at those as well, could give better results.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hello, @Marvin. It's a regression task actually. That's why I'm going with R2. At I thought of using sample mechanisms, but I was asked to try to develop a neural network out of the data, but I'll sure try other models as well. I'll take a look at Keras. Thanks $\endgroup$ – Lucas Abreu Sep 23 '19 at 10:42
  • $\begingroup$ The data is actually parameters from another model. Basically, the signal comes from a signal read from chemicals which had its signal modelled by an equation for classification tasks. Now I'm trying to use the parameters of the model to predict other properties from the chemicals $\endgroup$ – Lucas Abreu Sep 23 '19 at 10:49
  • $\begingroup$ If its that have you thought about looking into performing a dimensionality reduction? Maybe a simple PCA or SVD reduces your features by enough to work. If not I think @Leevo gave a good answer, using Trees to estimate importance. In general tho I would never run neural networks in SciKit, the implementation is really bad :). $\endgroup$ – Marvin Purtorab Sep 23 '19 at 13:44
  • $\begingroup$ Have already gave a try and didn't quite work. Do you recommend a tutorial for Keras? I'd have to check a couple of hyperparameters of the network $\endgroup$ – Lucas Abreu Sep 23 '19 at 14:32
  • $\begingroup$ Yes sure, tensorflow.org/tutorials/keras/basic_classification should probably do what you want. The difference between classification and regression is only (essentially) the loss you want to optimize (probably either RMSE or MAE) and that your output layer would only have 1 neuron as output, as you are only predicting a single number. Another approach to check for feature importance can be both things like LIME or SHAP values, both arent super trivial to implement though, even with packages. $\endgroup$ – Marvin Purtorab Sep 23 '19 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.