# Linked Questions

3answers
23k views

### Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
1answer
32k views

### Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
2answers
27k views

### Why my training and validation loss is not changing?

I used MSE loss function, SGD optimization: ...
2answers
6k views

### How to check for dead relu neurons

Background: While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as ...
2answers
2k views

### Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0?

By definition, Relu is max(0,f(x)). Then its gradient is defined as: 1 if x > 0 and 0 if x < 0. Wouldn't this mean the ...
1answer
287 views

### Why do CNNs with ReLU learn that well?

Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU): $$f(x) = max(0, x)$$ However, the derivative of this function is f'(x) = \begin{cases} 0 &...
3answers
304 views

### If ReLU is so close to being linear, why does it perform much better than a linear function?

ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers. I'm a beginner to deep learning research and methodologies but I've already seen several ...
2answers
409 views

### Why is the "dying ReLU" problem not present in most modern deep learning architectures?

The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks. However it has been shown that it can suffer from the dying Relu problem (see also What is the "dying ...
0answers
227 views

### Bias of 1 in fully connected layers introduced dying relu problem

While implementing AlexNet (model-code), one of the thing I need to do was to initialize the biases of the convolutional layers and fully connected layers. Normally we initialize biases with 0s, but ...
1answer
42 views

### Vanishing gradient problem even after existence of ReLu function?

Let's say I have a deep neural network with 50 hidden layers and at each neuron of hidden layer the ReLu activation function is used. My question is Is it possible for vanishing gradient problem to ...