When is a neural network better “traditional” models like decisions trees and lassos?

There's a whole theory of statistical inference based off calculus studying consistency, efficiency, robustness, BLUE, unbiasedness of linear models (Gaussian,Exponential, Chi-square, F-distribution, etc)... that make up regression models.

When is a neural network better than these traditional models based off calculus that are used regression-analysis? Is there a whole mathematical theory including robustness, Rao-blackwell stuff, consistency, sufficency dedicated to them that someone like Wackerly/HOFF writes about in a book?

Are neural networks not explainable by calculus as logistic regression is explainable as the maximum likelihood estimator?

What's the mathematics behind neural networks or is that more CompSci theory?

Are neural networks unbiased estimators that are consistent, efficient, etc that are on average the right answer? Can they estimate the true parameters of the population?

1 Answer

Neural network are "just" a way of estimating a numerical function. Their power is that this comes out of simple matrix multiplications with non linear functions that can be efficiently parallelized. As such, they are perfectly described by calculus.

The question on whether they are biased or not is then related to how they are trained and their black-box behavior. You can put stats at the end of a neural network as well.

The problem IMHO is that generalization for them is a complex topic. Presented with new data, they can have unexpected behavior compared to more traditional techniques for which we understand the behavior at the edge.

If you consider simple regression, like least squares, then neural networks are not actually in play. We use the frameworks to build equations, but we don't build neural networks. As such, we just benefit from the capability of these to process huge amount of data.

We are still going to estimate $$y=ax+b$$ if we want to estimate the coefficients. For logistic regression, it's still the same boundaries as if we were doing a logistic regression without neural networks.

Neural networks are only in play when the function is unknown and needs to be estimated.