How do neural networks account for outliers?

Question

How do neural networks account for outliers and overfitting?

Answer 1

Here's a math answer for you.

Neural network is an approximation function $f(\theta)$ of the joint distribution $p(X, Y)$ of input data $X$ and labels $Y$. The learning process is the process of tweaking parameters $\theta$ to make $f$ as close as possible to $p$

$$f(\theta) \approx p(X, Y)$$

^{Side note: usually $f$ is considered to approximate the conditional $p(Y|X)$, but it can be viewed more generally.}

So, in this terms, outliers are the values $(x, y)$ clearly outside of the distribution $p(X, Y)$. Hence, a neural network can not account for them, this has to be done separately. One possibility is to gather more data to make these outliers look ordinary, i.e., from $p(X, Y)$.

As for overfitting, it's completely different term, which relates to inability to generalize. This happens because in practice the true distribution $p(X, Y)$ is never fully known. Instead, the researches have the sample of it -- $(\hat{X}, \hat{Y})$, a.k.a. the training data. If the dimensionality of $\theta$ is large, $f$ can approximate $p(\hat{X}, \hat{Y})$ so well that it actually learns the noise and fails to capture $p(X, Y)$. There are ways to deal with overfitting, most importantly regularization, which is equivalent to adding noise to $(\hat{X}, \hat{Y})$. So the answer to this question is: it can be extended to account for it, by using special techniques. But if the researcher does nothing, the neural network won't learn to deal with overfitting by itself.