An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation function is needed to apply only once, on 1 final layer.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
Not only for Convolutional Neural Networks (CNNs), also for DNNs (Deep Neural Networks) and RNNs (Recurrent Neural Networks), we use activation functions at every layer. Sigmoid (for binary classification), softmax (for multiclass classification) or some other types are usually used at the final output layer, each specific for the kind of labels that we have to compare with the predictions.
However, other neurons require activation functions as well, especially for nonlinearity purposes; most popular ones are ReLUs(Rectified Linear Units), Leaky ReLU, tanh,.. and so on. We nearly always use an activation function for every neuron in Deep Learning. For a detailed insight, have a look at:
https://www.youtube.com/watch?v=Xvg00QnyaIY
Also specificly for Convolutional Nets:
https://www.youtube.com/watch?v=jPOAS7uCODQ&list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF&index=7
Hope I could help, please do not hesitate to ask more. Good Luck!
-
$\begingroup$ thx. an activation function on the final layer is a must, while i think it's ok to not use an activation function on previous layers if i dont find necessary to use batch normalization, pooling etc? $\endgroup$– feynmanCommented Feb 19, 2019 at 2:38
-
$\begingroup$ You need activation function at previous layers, the activation functions are the stuff that make your model learn some strong patterns by their non-linear nature. If you use a deep learning library; you may not need to specify the activation function for layers. However, the network you use will still have activation functions which are default by the library, most probably 'linear' or 'relu'. $\endgroup$ Commented Feb 19, 2019 at 19:44
-
$\begingroup$ if 'linear' then there's actually no activation function? anyway i write my own code, so i only use an activation when necessary $\endgroup$– feynmanCommented Feb 20, 2019 at 1:41
-
1$\begingroup$ Not specifying an activation code would mean that you use a linear function g(z) = z in the literature of Deep Learning. That's ,of course, your choice but just know that your algorithm will lose its 'magic' without nonlinear activation functions all over the place. Good luck! $\endgroup$ Commented Feb 20, 2019 at 9:57
$\begingroup$
$\endgroup$
Training a network is analogous to fitting a function to some scalar data. If the data is linear, fitting a linear function is appropriate and will work well.
In the case of deep learning, the data is rich and non-linear, so we apply non-linear activation functions to make the model more complex.
Another reason we use activation functions on intermediate layers is to keep the weight and output values close to 0 and "kind of" Gaussian, for optimization reasons.
