How to marry the fact that (most) neural networks with a single hidden layer are universal function approximators with the fact that in the bias-variance decomposition we consider there to be an irreducible error?

Can the irreducible error be 0? (Then it would be inappropriate named.)


The answer to the title question is "no"; the contexts are a little different.

The bias-variance tradeoff holds in the context that $y=f(X)+\varepsilon$, with $\varepsilon$ having mean 0 and variance $\sigma^2$; and $\sigma^2$ turns out to be the irreducible error, so named because it is a lower bound on the expected squared-error. So to answer your last question, the irreducible error can be zero, if there is no noise in the data.

That neural networks are universal function approximators refers to the actual output function itself (many linear functions with, say, ReLU activators), and not to the fitting procedure. Notice too that the universality has restrictions: the approximation is only guaranteed on a compact domain, and the error in approximation can be pushed down by using enough neurons in the hidden layer.

So what the universality really implies, w.r.t. the bias-variance tradeoff, is that it is possible to push the bias to zero (or as close as desired, by using enough neurons). Of course, this will generally result in a badly overfit model.

wikipedia - Bias-variance decomposition


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.