2
$\begingroup$

I've seen at many places that sometimes neural networks simply memorizes training data. What that means actually ?

Neural network consists of bunch of weights which gets trained and outputs based on input data. It'll output different thing for different input. Where does this memorization comes into play ?

$\endgroup$
3
$\begingroup$

Memorization is the same as overfitting. The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.

In order to understand what I mean by overfitting and memorization let us look at the polynomial regression

$$y_n=w_0+w_1x_n+w_2x_n^2+ \varepsilon.$$

We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients. We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.

| improve this answer | |
$\endgroup$
1
$\begingroup$

If you have a very complex machine learning models (i.e. one with a lot of parameters) and you try to train it on a fairly small dataset (i.e. few samples), then the model has the capability of memorizing those samples. This means that it will learn a set of weights where for every single one of the input samples, it will predict its label exactly! This is apparent beacause the model reaches a training loss of zero.

This is referred to as overfitting and is a problem because, while the model is performing adequately on the training set, it can't generalize on unseen data.

If you want to read on how to prevent this I suggest reading this post.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.