Memorization is the same as overfitting. The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.
In order to understand what I mean by overfitting and memorization let us look at the polynomial regression
We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients.
We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.