What are the activation functions in Convolutional Layers for?

Question

I read a lot about CNNs but I didn't quite understand some things:

1. What are the activation function in CLayers for? If I understood it right, the only weights in these layers are the ones in Filters, and for the activation function a weighted sum is needed?

2. The computational effort should increase, shouldn't it? When there are many Filters many Feature Maps (Map with the dot-products) are produced. All of them are given to the next Layer, so if they are as big as the input Image and there are 10 Filters in the first CLayer the second CLayer would have to apply 10x the computational effort(per Filter in the second Layer) that the first layer had or, since they all take all the last layers outputs (the feature maps).

3. If there are more Layers than one, how does the backpropagation know what they have to change on each one of them, especially since the backpropagation occurs after all layers are applied.

(CLayer = Convolutional Layer)

Answer 1

1 - Activation functions are non-linear functions. These are added in between layers which are simply Linear transformations.

Example without activation function:

ConvLayer1(Input) -> ConvMaps1 
ConvLayer2(ConvMaps2) -> ConvMaps2

Mathematically, this would be $I_{nput} \circledast K_{ernel_1} \circledast K_{ernel_2} $,

which is equivalent to $I_{nput} \circledast K_{ernel_3} $,

where $K_{ernel_3} = K_{ernel_1} \circledast K_{ernel_2} $

That means that creating two layers would simply waste computational effort and give no gain whatsoever.

2 - Yes, if you keep the same image Width and Heigh and just stack more channels, there convolution maps will grow out of control, this is why you compensate this with Pooling Operations and use of large strides. Also, increasing the number of parameters without proper care and architecture design usually leads to overfitting.

3 - Backpropagation is simply a computationally efficient and convenient way of expressing the Partial Derivatives of the function and acts exactly as it would if you just took the derivative of the whole function at once. There is plenty of content online on it, you probably didn't understand the process of weight updating.

Answer 2

1. Why activation functions: For this , its straight forward to add some non-linearity into the model, and this helps in defining which neuron to fire which should not.
2. For second one , the compute will increase slightly but not so rapidly , why because the convolution operation you are performing on the first layer will reduce the image size in the second layer. for example , you have a 32 X 32 image and you use a 5 X 5 kernel , you will end up getting 28 X 28 feature maps. Hence going into deeper layers the size will be reduced.
3. For backpropagation , this like this, you have input to the last layer (from last-1 layer) and you have weight matrix in the last layer. You will predict the output in last layer, and calculate the loss. Here you will be able to calculate the loss with respect to weights and also loss with respect to the input. Hence the gradient that comes would be send to the previous layers for weight updates.