First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type of activators only at the first layers of the CNN and afterwards a simple RELU follows at the end. Regarding this, how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?
Secondly, as far as I'm concerned because of the vanishing gradient problem the neurons at the beginning tend to fall into zero more often than those at the top of the network and then it is very difficult (or even impossible) to activate again; Wouldn't be correct to allow the negative gradient at the whole pipeline of the Neural Network?