Linked Questions

37
votes
3answers
23k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
19
votes
1answer
32k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
8
votes
2answers
27k views

Why my training and validation loss is not changing?

I used MSE loss function, SGD optimization: ...
13
votes
2answers
6k views

How to check for dead relu neurons

Background: While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as ...
10
votes
2answers
2k views

Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0?

By definition, Relu is max(0,f(x)). Then its gradient is defined as: 1 if x > 0 and 0 if x < 0. Wouldn't this mean the ...
4
votes
1answer
287 views

Why do CNNs with ReLU learn that well?

Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU): $$f(x) = max(0, x)$$ However, the derivative of this function is $$f'(x) = \begin{cases} 0 &...
2
votes
3answers
304 views

If ReLU is so close to being linear, why does it perform much better than a linear function?

ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers. I'm a beginner to deep learning research and methodologies but I've already seen several ...
3
votes
2answers
409 views

Why is the "dying ReLU" problem not present in most modern deep learning architectures?

The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks. However it has been shown that it can suffer from the dying Relu problem (see also What is the "dying ...
1
vote
0answers
227 views

Bias of 1 in fully connected layers introduced dying relu problem

While implementing AlexNet (model-code), one of the thing I need to do was to initialize the biases of the convolutional layers and fully connected layers. Normally we initialize biases with 0s, but ...
1
vote
1answer
42 views

Vanishing gradient problem even after existence of ReLu function?

Let's say I have a deep neural network with 50 hidden layers and at each neuron of hidden layer the ReLu activation function is used. My question is Is it possible for vanishing gradient problem to ...