I am doing some modeling to predict a variable of interest given a big set of features (500) for which I expect a considerable amount of interactions happening at least among some of them.

I first tried to do a multilinear regression and the predictions were not so good (R2 around 0.2) but then I switched to Random forests and my R2 got much better (0.5/0.7 test/train set). I thought by using more complex models I could get an even better performance, but in reality, this does not happen.

I tried a vanilla FNN after standardizing all features, with different setups:

  • If I set 2 hidden layers it gets overtrained very easily (R2= 0/0.8 test/train set)
  • If I set one hidden layer and a reasonably high number of nodes it still gets overtrained but less badly (R2 = 0.2/0.8 test/train set)
  • I also tried increasing a lot the number of epochs (up to 1000) and it didn't help at all.

So for some reason, FNN cannot outperform the random forest in this case.

Do you have any suggestions for a good choice of model for a case like this?
Some relevant info are the following

  • Most features are coordinates (define the position of the object in the space)
  • They are all numeric features, some are continuous and some are discrete
  • I expect that there is a lot of interactions between some features
  • Some feature are probably useless (according to the RF model)
  • In my case the best I can expect is probably R2 = 0.8

If you need more info comment below and i will add those in the questions.

More Info

  • I have 6400 observations not time related
  • Each observation is basically a measure of a pair of 2 objects. That means that my response variable is not a trait of one simple object but can be measured only between pairs of objects
  • $\begingroup$ Regarding the FNN, you should normalize the features in a given range, e.g. [-1, 1], then regularize the network (try weight decay first, then dropout). Moreover, you can also try an xgboost model, which I expect to beat the RF. $\endgroup$ Dec 7, 2023 at 21:32
  • $\begingroup$ Yes i did standardize all the features for the FNN, i forgot to mention that before but now I've added it. $\endgroup$
    – Mirko
    Dec 8, 2023 at 0:53
  • $\begingroup$ How many observations do you have? Is your data a time series? $\endgroup$ Dec 8, 2023 at 9:03
  • $\begingroup$ Added more information on the observation number and type $\endgroup$
    – Mirko
    Dec 8, 2023 at 17:54
  • $\begingroup$ @Mirko you can try factorization machines. $\endgroup$
    – noe
    Dec 9, 2023 at 14:44

2 Answers 2


From my experience, as you model physical phenomena, neural networks should outperform any tree-based method. I don't even mean any error metric, but the issue is that the neural network's prediction might be smooth and the tree's not - it always will have a "steps-shaped" nature. If the quality of interpolation and modelling an actual phenomenon is important to you, you should stick to a neural network, in my view.

However, relationships in your dataset might be pretty complex and vanilla NN might not be sufficient. For me is important to set such activation functions in your neural network to mirror relationships between input variables and the output variable. In a perfect case, if you did it well, it would even facilitate your model's extrapolation capabilities. You should start with the analysis of your data. You can start from:

  • remove insignificant variables - if the random forest finds some variable unnecessary, it probably is
  • handle highly correlated input variables - if a few variables have the same information, just one of them might be enough
  • identify the most vital variables and visualise how they correlate with the output variable - it will help you identify what type of activation function you should use
  • consider transforming some variables to help you predict better - it could be a difference or product of some features, try to understand them more thoroughly and apply this knowledge. Sometimes even changing a unit helps a lot.

Besides, I can give you a few pieces of advice on how to build your network better:

  • use regularization to generalise better - e.g. Dropout
  • adjust the depth of your network - in many cases one hidden layer is not enough
  • adjust the width of your network - it doesn't have to be very wide
  • activation might be the key - I repeat
  • remember to scale your data - BatchNormalization layer at the very beginning might be sufficient

I hope something from this list will be helpful for you.


Try different regularization techniques in FNN to help it generalize better.

Gradient-boosted regressors work well too.

Maybe a piecewise multilinear regression could work.

Not experienced in this but Bayesian models could deal with uncertainty better.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.