Let's say I have a deep neural network with 50 hidden layers and at each neuron of hidden layer the ReLu activation function is used. My question is

  • Is it possible for vanishing gradient problem to get occur during the backpropogation for weights updates even after the existence of relu?
  • or we can say that vanishing gradient problem will never occur when all the activation functions are ReLu?"

Are you talking about LeakyReLU by chance and not ReLU? Because ReLU is known for vanishing gradients, since any values less than zero are mapped to zero. This is true regardless of the number of layers. LeakyReLU on the other hand, maps the values less than zeros to a very small positive number. This prevents vanishing gradient from occurring.

EDIT: LeakyReLU prevents dying ReLU from occurring not vanishing gradient. PReLU prevents vanishing gradient from occurring.

EDIT 2: To answer the comments. VGG was proposed in the paper "Very Deep Convolutional Networks for Large-Scale Image Recognition" and was one of the top performance models for the ImageNet challenge at its time. Architecture wise, VGG wasn't completely different from what was done in the past. However, it was much deeper. This is impart where the vanishing gradient becomes a problem. It does not really have to do with ReLU alone but a combination of every single layer.

Enter ResNet, which uses skip connections. These actually make parts of the networks shallow and makes it easier for the network to learn both easy and difficult tasks (ie. low and high frequencies in images). More difficult tasks require more learnable parameters while easier tasks require fewer parameters.

I believe PReLU being a learnable activation function can help deal with this task.

  • $\begingroup$ This is not correct. The leaky ReLU is a fix for "dying ReLU" - see datascience.stackexchange.com/questions/5706/… - and it does not address vanishing gradient problems significantly better than ReLU (there may be a small improvement proportional to the gradient below zero input) $\endgroup$ Jul 19 at 17:29
  • $\begingroup$ Yeah, my bad I got the two confused. To answer the question I think PreLU then fixes the vanishing gradient problem. stats.stackexchange.com/questions/279209/… $\endgroup$
    – J Houseman
    Jul 19 at 17:56
  • $\begingroup$ @NeilSlater So, why does residual network architecture like Resnet considered as best gradient superhighways as compared to architecture like VGG16? If VGG16 do have relu activation function than why their performance degrades?Is it possible that DNN do have vanishing gradient problem even after using relu activation function? Please help me to clear this. $\endgroup$
    – Bits
    Jul 19 at 17:59
  • $\begingroup$ I know less about PReLU, but again I don't think it is intended as a fix for vanishing gradient (any more than ReLU already is). Residual networks as mentioned by the OP, are likely more effective. $\endgroup$ Jul 19 at 19:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.