# What it means for a neural network to memorize data

I've seen at many places that sometimes neural networks simply memorizes training data. What that means actually ?

Neural network consists of bunch of weights which gets trained and outputs based on input data. It'll output different thing for different input. Where does this memorization comes into play ?

Memorization is the same as overfitting. The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.

In order to understand what I mean by overfitting and memorization let us look at the polynomial regression

$$y_n=w_0+w_1x_n+w_2x_n^2+ \varepsilon.$$

We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients. We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.

If you have a very complex machine learning models (i.e. one with a lot of parameters) and you try to train it on a fairly small dataset (i.e. few samples), then the model has the capability of memorizing those samples. This means that it will learn a set of weights where for every single one of the input samples, it will predict its label exactly! This is apparent beacause the model reaches a training loss of zero.

This is referred to as overfitting and is a problem because, while the model is performing adequately on the training set, it can't generalize on unseen data.

If you want to read on how to prevent this I suggest reading this post.