2
$\begingroup$

An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation function is needed to apply only once, on 1 final layer.

$\endgroup$
1
$\begingroup$

Not only for Convolutional Neural Networks (CNNs), also for DNNs (Deep Neural Networks) and RNNs (Recurrent Neural Networks), we use activation functions at every layer. Sigmoid (for binary classification), softmax (for multiclass classification) or some other types are usually used at the final output layer, each specific for the kind of labels that we have to compare with the predictions.

However, other neurons require activation functions as well, especially for nonlinearity purposes; most popular ones are ReLUs(Rectified Linear Units), Leaky ReLU, tanh,.. and so on. We nearly always use an activation function for every neuron in Deep Learning. For a detailed insight, have a look at:

https://www.youtube.com/watch?v=Xvg00QnyaIY

Also specificly for Convolutional Nets:

https://www.youtube.com/watch?v=jPOAS7uCODQ&list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF&index=7

Hope I could help, please do not hesitate to ask more. Good Luck!

$\endgroup$
4
  • $\begingroup$ thx. an activation function on the final layer is a must, while i think it's ok to not use an activation function on previous layers if i dont find necessary to use batch normalization, pooling etc? $\endgroup$
    – feynman
    Feb 19 '19 at 2:38
  • $\begingroup$ You need activation function at previous layers, the activation functions are the stuff that make your model learn some strong patterns by their non-linear nature. If you use a deep learning library; you may not need to specify the activation function for layers. However, the network you use will still have activation functions which are default by the library, most probably 'linear' or 'relu'. $\endgroup$
    – Ugur MULUK
    Feb 19 '19 at 19:44
  • $\begingroup$ if 'linear' then there's actually no activation function? anyway i write my own code, so i only use an activation when necessary $\endgroup$
    – feynman
    Feb 20 '19 at 1:41
  • 1
    $\begingroup$ Not specifying an activation code would mean that you use a linear function g(z) = z in the literature of Deep Learning. That's ,of course, your choice but just know that your algorithm will lose its 'magic' without nonlinear activation functions all over the place. Good luck! $\endgroup$
    – Ugur MULUK
    Feb 20 '19 at 9:57
1
$\begingroup$

Training a network is analogous to fitting a function to some scalar data. If the data is linear, fitting a linear function is appropriate and will work well.

In the case of deep learning, the data is rich and non-linear, so we apply non-linear activation functions to make the model more complex.

Another reason we use activation functions on intermediate layers is to keep the weight and output values close to 0 and "kind of" Gaussian, for optimization reasons.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.