1
$\begingroup$

In the lecture notes of CS231n, it says (emphasis mine)

... There are three major sources of memory to keep track of:

  • From the intermediate volume sizes: These are the raw number of activations at every layer of the ConvNet, and also their gradients (of equal size). Usually, most of the activations are on the earlier layers of a ConvNet (i.e. first Conv Layers). These are kept around because they are needed for backpropagation, but a clever implementation that runs a ConvNet only at test time could in principle reduce this by a huge amount, by only storing the current activations at any layer and discarding the previous activations on layers below.
  • ......

Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?

I guess output is the top layer. This is because we only need the gradients of the next (closer to output) layer to compute those of the current layer. The exact activations of the next layer are not required, so we can discard them after the corresponding gradients have been computed. This is possible because we compute gradients backwards (last layers first, first layers last).

$\endgroup$
2
$\begingroup$

Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?

Yes, a "below" layer means an "earlier" layer, i.e. closer to the input. For CNNs it is quite common to speak of lower layers, lower level feature maps etc.

And for RNNs you can even take this literally as they are usually visualized bottom-to-top instead of left-to-right (e.g. see figures here).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.