1
$\begingroup$

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type of activators only at the first layers of the CNN and afterwards a simple RELU follows at the end. Regarding this, how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

Secondly, as far as I'm concerned because of the vanishing gradient problem the neurons at the beginning tend to fall into zero more often than those at the top of the network and then it is very difficult (or even impossible) to activate again; Wouldn't be correct to allow the negative gradient at the whole pipeline of the Neural Network?

Improve this question
$\endgroup$
2
  • $\begingroup$ I write a paper about lrelu and relu and found your post very interesting. Can you help me about 1 question? You say: "However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type of activators only at the first layers of the CNN" --> can you please cite at least one paper which does that? I searched a lot, but still cannot find anything. Thank you ver much. $\endgroup$
    – Manolis Pintelas
    Dec 10, 2021 at 3:17
  • $\begingroup$ Hi @ManolisPintelas, this approach is being used mostly with GANs. I think I noticed the same approach with YOLO object detectors v3/4. I don't have a paper to cite here for you. But if you check the implementations there you can find this approach. However, I haven't found any explanation of that. The accepted answer is the best explanation found honestly. All these tricks are used by experiment mostly... $\endgroup$
    – Dimimal13
    Dec 16, 2021 at 12:59

1 Answer 1

Reset to default
0
$\begingroup$

how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

What matters is to add a non-linearity to the outputs of a neuron. Consequently, by employing each function that adds non-linearity, the network will work due to the derivative that can backpropagate the differentiation of the error term. If you use Relu or leaky Relu, the update terms will change but your network will work fine. The reason that leaky version is used in the mentioned papers is due to avoiding dying relu problem which can happen a lot for regression problems.

About your second question, consider that it usually suffice to use leaky relu in the first layers and due to them, the chance of deep neurons to be stuck at zero is not very much as the results of those papers show. You can use leaky version all over the network but by experience, relu is very fast to be trained!

Improve this answer
$\endgroup$
2
  • $\begingroup$ So, dying relu problem most likely occurs in regression problems than classification? Leaky activation functions adds significant overhead in training? $\endgroup$
    – Dimimal13
    Nov 18, 2018 at 18:35
  • $\begingroup$ Dying relu may happen for both cases. In my own experience, I've seen that alot in regression problems. Speaking about which happens more needs a paper that I've not seen yet but there may be. Leaky activations are usually used with small slope. Again I can't say it adds a siginificant overhead. It depends on the problem in hand. $\endgroup$
    – Green Falcon
    Nov 18, 2018 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.