In the lecture notes of CS231n, it says (emphasis mine)
... There are three major sources of memory to keep track of:
- From the intermediate volume sizes: These are the raw number of activations at every layer of the ConvNet, and also their gradients (of equal size). Usually, most of the activations are on the earlier layers of a ConvNet (i.e. first Conv Layers). These are kept around because they are needed for backpropagation, but a clever implementation that runs a ConvNet only at test time could in principle reduce this by a huge amount, by only storing the current activations at any layer and discarding the previous activations on layers below.
Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?
I guess output is the top layer. This is because we only need the gradients of the next (closer to output) layer to compute those of the current layer. The exact activations of the next layer are not required, so we can discard them after the corresponding gradients have been computed. This is possible because we compute gradients backwards (last layers first, first layers last).