Skip to main content
58 votes
Accepted

Why is ReLU used as an activation function?

In mathematics (linear algebra) a function is considered linear whenever a function$f: A \rightarrow B$ if for every $x$ and $y$ in the domain $A$ has the following property: $f(x) + f(y) = f(x+y)$. ...
Tophat's user avatar
  • 2,430
42 votes
Accepted

What is GELU activation?

GELU function We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}...
Esmailian's user avatar
  • 9,342
31 votes
Accepted

Why ReLU is better than the other activation functions

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by ...
Maxim's user avatar
  • 900
26 votes
Accepted

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer. Import the LeakyReLU and instantiate a ...
n1k31t4's user avatar
  • 14.9k
19 votes
Accepted

Difference of Activation Functions in Neural Networks in general

A similar question was asked on CV: Comprehensive list of activation functions in neural networks with pros/cons. I copy below one of the answers: One such a list, though not much exhaustive: ...
Franck Dernoncourt's user avatar
18 votes

How to create custom Activation functions in Keras / TensorFlow?

First you need to define a function using backend functions. As an example, here is how I implemented the swish activation function: ...
Simon Larsson's user avatar
13 votes

Why is ReLU used as an activation function?

I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation. This is not completely true. The neurons are not dead. If you use sigmoid-like activations, after some ...
Green Falcon's user avatar
  • 14.1k
12 votes

How does Sigmoid activation work in multi-class classification problems

If your task is a kind of classification that the labels are mutually exclusive, each input just has one label, you have to use Softmax. If the inputs of your ...
Green Falcon's user avatar
  • 14.1k
11 votes

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero, because each derivative could be potentially larger than one (up to $n$). However, for practical ...
Esmailian's user avatar
  • 9,342
11 votes

Neural Network not Deep

The term 'deep' in deep learning is not well defined, but is generally meant to apply to networks that have stacked multiple layers on top of each other to create a deep network. It is therefore up ...
Oxbowerce's user avatar
  • 7,602
10 votes
Accepted

Input normalization for ReLu?

You have to normalize your data to accelerate learning process but based on experience its better to normalize your data in the standard manner, mean zero and standard deviation one. Although mapping ...
Green Falcon's user avatar
  • 14.1k
10 votes
Accepted

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you. Why do so many functions used in data science ...
Maxim's user avatar
  • 900
10 votes
Accepted

What is the purpose of multiple neurons in a hidden layer?

To explain using the sample neural network you have provided: Purpose of the multiple inputs: Each input represents a feature of the input dataset. Purpose of the hidden layer: Each neuron learns a ...
Sandeep S. Sandhu's user avatar
10 votes

What is GELU activation?

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show ...
BookYourLuck's user avatar
10 votes
Accepted

Why deep learning models still use RELU instead of SELU, as their activation function?

ReLU is quick to compute, and also easy to understand and explain. But I think people mainly use ReLU because everyone else does. The activation function doesn't make that much of a difference, and ...
Darren Cook's user avatar
  • 1,074
8 votes

Activation function vs Squashing function

Activation functions like sigmoid function, hyperbolic tangent function, etc. are also called squashing function because they squash the input into a small range like in sigmoid function output is in ...
Rajat Gupta's user avatar
8 votes

Why is activation needed at all in neural network?

Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and ...
matthiaw91's user avatar
  • 1,545
8 votes
Accepted

What is the advantage of using Euler's number ($e^x$) instead of another base in the softmax equation?

Choosing a different base would squash the graph of the function uniformly in the horizontal direction, since $$ a^x = e^{x\cdot \ln(a)}. $$ The exponential function with base $e$ is widely considered ...
Amirhossein Rezaei's user avatar
7 votes

How does Sigmoid activation work in multi-class classification problems

softmax() will give you the probability distribution which means all output will sum to 1. While, sigmoid() will make sure the output value of neuron is between 0 to 1. In case of digit ...
Preet's user avatar
  • 638
6 votes
Accepted

Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially ...
Neil Slater's user avatar
6 votes
Accepted

Activation function vs Squashing function

An activation function This the name given to a function, which is applied to a neuron that just had a weight update as a result of new information. It can refer to any of the well known activation ...
n1k31t4's user avatar
  • 14.9k
6 votes

What does it mean for an activation function to be "saturated/non-saturated"?

Please see this answer. An activation function is considered non-satured if $$ \lim_{z \rightarrow \infty} f(z) = \infty $$ A saturated activation function has a compact range such as $[-1,1]$ for $...
TheDataScienceNinja's user avatar
6 votes

How batch normalization layer resolve the vanishing gradient problem?

Batch Normalization (BN) does not prevent the vanishing or exploding gradient problem in a sense that these are impossible. Rather it reduces the probability for these to occur. Accordingly, the ...
Jonathan's user avatar
  • 5,430
5 votes

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

I believe the question was about using LeayReLU within the Keras Functional API. Which would look something like this: ...
WaveRider's user avatar
  • 151
5 votes

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

Just to add another solution. You can also write something like import tensorflow as tf keras = tf.keras ...
Somoy Subandhu's user avatar
5 votes
Accepted

What does the descision boundary of a relu look like?

What would be the shape of this separator (decision boundary) in case we also take a relu on the output and only then threshold? For just a single neuron, indeed the decision boundary will just be a ...
Ben Reiniger's user avatar
  • 11.9k
5 votes
Accepted

Difference between ReLU, ELU and Leaky ReLU. Their pros and cons majorly

Look at this ML glossary: ELU ELU is very similiar to RELU except negative inputs. They are both in identity function form for non-negative inputs. On the other hand, ELU becomes smooth slowly ...
OmG's user avatar
  • 1,219
5 votes

How to prove Softmax Numerical Stability?

Take into accout that $e^{a-b} = e^a \cdot e^{-b}$, therefore: $\dfrac{e^{s_{y_i} - s_{max}}}{\sum e^{s_k - s_{max}}} = \dfrac{e^{s_{y_i}} e^{- s_{max}}}{\sum e^{s_k} e^{- s_{max}}} = \dfrac{e^{s_{y_i}...
noe's user avatar
  • 27k
4 votes
Accepted

Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

In certain network structures having symmetric activation layers has advantages (certain autoencoders for example) In certain scenarios having an activation with mean 0 is important (so tanh makes ...
Jan van der Vegt's user avatar
4 votes

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

I don't know of any papers about this topic, but intuitively it makes a lot of sense to use monotonic activation functions. Let's say we have a non-monotonic activation function, maybe a Gaussian ...
Jan van der Vegt's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible