I am making neural networks of multiple targets, all using same training data. For some of these targets, multivariate linear regressions do a very good job, i.e. a strong linear relation exists between training data and a particular target, whilst for some targets multivariate linear regressions are v poor predictors.

When I make a neural networks for some targets that regressions are v poor at, I can get an improved prediction accuracy/lower loss than with multivariate linear regression. However, for a few targets where there's a strong linear relationship, I cannot get neural networks to equal that simple multivariate linear regression performance, or beat them.

I would have thought that neural networks would get at least the same MSE as for linear regressions for a few targets. Is it always the case that neural networks will capture (and equal or better) simple linear relationships?

If I have x_1,...,x_m features, then multivariate regression does (w * x_1) + ... (w * x_m) + b

I have tried training neural networks with m nodes and e.g. 2 layers, so they are not much more complicated than regressions. I would have thought they find this easy.

So is there anything obvious I could be doing wrong?

  • $\begingroup$ Overfitting is a good candidate. We could probably help more if you add more information, like the loss curves for the training and validation sets. $\endgroup$
    – noe
    Jul 5, 2022 at 21:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.