I am making neural networks of multiple targets, all using same training data. For some of these targets, multivariate linear regressions do a very good job, i.e. a strong linear relation exists between training data and a particular target, whilst for some targets multivariate linear regressions are v poor predictors.
When I make a neural networks for some targets that regressions are v poor at, I can get an improved prediction accuracy/lower loss than with multivariate linear regression. However, for a few targets where there's a strong linear relationship, I cannot get neural networks to equal that simple multivariate linear regression performance, or beat them.
I would have thought that neural networks would get at least the same MSE as for linear regressions for a few targets. Is it always the case that neural networks will capture (and equal or better) simple linear relationships?
If I have x_1,...,x_m features, then multivariate regression does (w * x_1) + ... (w * x_m) + b
I have tried training neural networks with m nodes and e.g. 2 layers, so they are not much more complicated than regressions. I would have thought they find this easy.
So is there anything obvious I could be doing wrong?