How do neural networks account for outliers and overfitting?


1 Answer 1


Here's a math answer for you.

Neural network is an approximation function $f(\theta)$ of the joint distribution $p(X, Y)$ of input data $X$ and labels $Y$. The learning process is the process of tweaking parameters $\theta$ to make $f$ as close as possible to $p$

$$f(\theta) \approx p(X, Y)$$

Side note: usually $f$ is considered to approximate the conditional $p(Y|X)$, but it can be viewed more generally.

So, in this terms, outliers are the values $(x, y)$ clearly outside of the distribution $p(X, Y)$. Hence, a neural network can not account for them, this has to be done separately. One possibility is to gather more data to make these outliers look ordinary, i.e., from $p(X, Y)$.

As for overfitting, it's completely different term, which relates to inability to generalize. This happens because in practice the true distribution $p(X, Y)$ is never fully known. Instead, the researches have the sample of it -- $(\hat{X}, \hat{Y})$, a.k.a. the training data. If the dimensionality of $\theta$ is large, $f$ can approximate $p(\hat{X}, \hat{Y})$ so well that it actually learns the noise and fails to capture $p(X, Y)$. There are ways to deal with overfitting, most importantly regularization, which is equivalent to adding noise to $(\hat{X}, \hat{Y})$. So the answer to this question is: it can be extended to account for it, by using special techniques. But if the researcher does nothing, the neural network won't learn to deal with overfitting by itself.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.