# Linked Questions

3answers
24k views

### Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
1answer
33k views

### Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
2answers
29k views

### Why my training and validation loss is not changing?

I used MSE loss function, SGD optimization: ...
2answers
7k views

### How to check for dead relu neurons

Background: While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as ...
2answers
2k views

### Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0?

By definition, Relu is max(0,f(x)). Then its gradient is defined as: 1 if x > 0 and 0 if x < 0. Wouldn't this mean the ...
2answers
581 views

### Why is the "dying ReLU" problem not present in most modern deep learning architectures?

The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks. However it has been shown that it can suffer from the dying Relu problem (see also What is the "dying ...
3answers
376 views

### If ReLU is so close to being linear, why does it perform much better than a linear function?

ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers. I'm a beginner to deep learning research and methodologies but I've already seen several ...
1answer
290 views

### Why do CNNs with ReLU learn that well?

Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU): $$f(x) = max(0, x)$$ However, the derivative of this function is f'(x) = \begin{cases} 0 &...
0answers
242 views

### Bias of 1 in fully connected layers introduced dying relu problem

While implementing AlexNet (model-code), one of the thing I need to do was to initialize the biases of the convolutional layers and fully connected layers. Normally we initialize biases with 0s, but ...
2answers
71 views

### Vanishing gradient problem even after existence of ReLu function?

Let's say I have a deep neural network with 50 hidden layers and at each neuron of hidden layer the ReLu activation function is used. My question is Is it possible for vanishing gradient problem to ...