23k views

### Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
32k views

### Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
27k views

### Why my training and validation loss is not changing?

I used MSE loss function, SGD optimization: ...
6k views

### How to check for dead relu neurons

Background: While fitting neural networks with relu activation, I found that sometimes the prediction becomes near constant. I believe that this is due to the relu neurons dieing during training as ...
2k views

### Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0?

By definition, Relu is max(0,f(x)). Then its gradient is defined as: 1 if x > 0 and 0 if x < 0. Wouldn't this mean the ...
287 views

### Why do CNNs with ReLU learn that well?

Convolutional Neural Networks (CNNs) use almost always the rectified linear activation function (ReLU): $$f(x) = max(0, x)$$ However, the derivative of this function is f'(x) = \begin{cases} 0 &...
304 views

### If ReLU is so close to being linear, why does it perform much better than a linear function?

ReLU is defined as being $x \mapsto x$ whenever $x \geq 0$ and is constant on zero for negative numbers. I'm a beginner to deep learning research and methodologies but I've already seen several ...
409 views

### Why is the "dying ReLU" problem not present in most modern deep learning architectures?

The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks. However it has been shown that it can suffer from the dying Relu problem (see also What is the "dying ...