On a CNN, what is use of using Activation function in convolution layer?
Does single weight is used for full matrix or for every pixel or box it may vary?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The use is the same as always, without the non-linear activation, no matter how many layers you use, the result will still be a linear model.
The non-linear activation is performed on every single output "pixel" indevidualy:
$\begingroup$
$\endgroup$
Activation function and a convolutional layer are generally separate things. It is just that they are usually used together and keras library has a parameter for activation that is in keras applied right after the convolution. Which is not always the best order of applying the functions together.
Activation function: We have an input data for the activation function. For every value in the input a function is applied. For example, we have 3 layers of 30x30 pixels (30x30x3). Activation function is applied to all 2700 values and returns the same dimensions. The result is 30x30x3. For example, we have Relu and the input is 0..255 values for RGB colors of the image. The output will not be different from the input because all the values are positive.
A convolutional layer can have the same input. For example, we apply 3x3 filter with depth 3 to all the data and we get 28x28x3 result. If we add a relu activation to a convolutional layer in keras it means that we take 28x28x3 as input and apply relu to all the values and get 28x28x3 values as output.
As mentioned in another answer, the activation function is the only thing that adds non-linearity. Without it the whole neural network is equal to a linear regression.
One of the best CNNs is ResNet from Microsoft. According to the latest papers activation applied before the convolution significantly improves the network and allows to increase the depth from 152 layer to a thousand layers. So the way keras applies activation is not the best way for ResNet.
