First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type of activators only at the first layers of the CNN and afterwards a simple RELU follows at the end. Regarding this, how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

Secondly, as far as I'm concerned because of the vanishing gradient problem the neurons at the beginning tend to fall into zero more often than those at the top of the network and then it is very difficult (or even impossible) to activate again; Wouldn't be correct to allow the negative gradient at the whole pipeline of the Neural Network?


how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

What matters is to add a non-linearity to the outputs of a neuron. Consequently, by employing each function that adds non-linearity, the network will work due to the derivative that can backpropagate the differentiation of the error term. If you use Relu or leaky Relu, the update terms will change but your network will work fine. The reason that leaky version is used in the mentioned papers is due to avoiding dying relu problem which can happen a lot for regression problems.

About your second question, consider that it usually suffice to use leaky relu in the first layers and due to them, the chance of deep neurons to be stuck at zero is not very much as the results of those papers show. You can use leaky version all over the network but by experience, relu is very fast to be trained!

| improve this answer | |
  • $\begingroup$ So, dying relu problem most likely occurs in regression problems than classification? Leaky activation functions adds significant overhead in training? $\endgroup$ – Dimimal13 Nov 18 '18 at 18:35
  • $\begingroup$ Dying relu may happen for both cases. In my own experience, I've seen that alot in regression problems. Speaking about which happens more needs a paper that I've not seen yet but there may be. Leaky activations are usually used with small slope. Again I can't say it adds a siginificant overhead. It depends on the problem in hand. $\endgroup$ – Media Nov 18 '18 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.