47 votes
Accepted

Why is ReLU used as an activation function?

In mathematics (linear algebra) a function is considered linear whenever a function$f: A \rightarrow B$ if for every $x$ and $y$ in the domain $A$ has the following property: $f(x) + f(y) = f(x+y)$. ...
user avatar
  • 2,187
39 votes
Accepted

What is GELU activation?

GELU function We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}...
user avatar
  • 8,479
27 votes
Accepted

Why ReLU is better than the other activation functions

The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by ...
user avatar
  • 830
22 votes
Accepted

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer. Import the LeakyReLU and instantiate a ...
user avatar
  • 13.8k
17 votes
Accepted

Difference of Activation Functions in Neural Networks in general

A similar question was asked on CV: Comprehensive list of activation functions in neural networks with pros/cons. I copy below one of the answers: One such a list, though not much exhaustive: ...
user avatar
14 votes

How to create custom Activation functions in Keras / TensorFlow?

First you need to define a function using backend functions. As an example, here is how I implemented the swish activation function: ...
user avatar
13 votes

Why is ReLU used as an activation function?

I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation. This is not completely true. The neurons are not dead. If you use sigmoid-like activations, after some ...
user avatar
  • 13.3k
11 votes

Neural Network not Deep

The term 'deep' in deep learning is not well defined, but is generally meant to apply to networks that have stacked multiple layers on top of each other to create a deep network. It is therefore up ...
user avatar
  • 5,641
10 votes
Accepted

What is the purpose of multiple neurons in a hidden layer?

To explain using the sample neural network you have provided: Purpose of the multiple inputs: Each input represents a feature of the input dataset. Purpose of the hidden layer: Each neuron learns a ...
user avatar
10 votes
Accepted

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you. Why do so many functions used in data science ...
user avatar
  • 830
10 votes

What is GELU activation?

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show ...
user avatar
9 votes
Accepted

Input normalization for ReLu?

You have to normalize your data to accelerate learning process but based on experience its better to normalize your data in the standard manner, mean zero and standard deviation one. Although mapping ...
user avatar
  • 13.3k
9 votes

How does Sigmoid activation work in multi-class classification problems

If your task is a kind of classification that the labels are mutually exclusive, each input just has one label, you have to use Softmax. If the inputs of your ...
user avatar
  • 13.3k
9 votes

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero, because each derivative could be potentially larger than one (up to $n$). However, for practical ...
user avatar
  • 8,479
8 votes

Why is activation needed at all in neural network?

Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and ...
user avatar
  • 1,485
5 votes
Accepted

Activation function vs Squashing function

An activation function This the name given to a function, which is applied to a neuron that just had a weight update as a result of new information. It can refer to any of the well known activation ...
user avatar
  • 13.8k
5 votes

Activation function vs Squashing function

Activation functions like sigmoid function, hyperbolic tangent function, etc. are also called squashing function because they squash the input into a small range like in sigmoid function output is in ...
user avatar
5 votes

How does Sigmoid activation work in multi-class classification problems

softmax() will give you the probability distribution which means all output will sum to 1. While, sigmoid() will make sure the output value of neuron is between 0 to 1. In case of digit ...
user avatar
  • 568
5 votes

What does it mean for an activation function to be "saturated/non-saturated"?

Please see this answer. An activation function is considered non-satured if $$ \lim_{z \rightarrow \infty} f(z) = \infty $$ A saturated activation function has a compact range such as $[-1,1]$ for $...
user avatar
4 votes
Accepted

Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

In certain network structures having symmetric activation layers has advantages (certain autoencoders for example) In certain scenarios having an activation with mean 0 is important (so tanh makes ...
user avatar
4 votes
Accepted

Why is an activation function notated as "g"?

The addition of the activation layer creates a composition of two functions. "A general function, to be defined for a particular context, is usually denoted by a single letter, most often the lower-...
user avatar
4 votes

Is classifier able to say there's no-such-case?

Taking your questions one after the other: Assume that I have a classifier which can classify left hand / right hand well. I am curious whether it can decide whether there's a hand in the image?...
user avatar
  • 7,996
4 votes

Why is ReLU used as an activation function?

The main reason to use an Activation Function in NN is to introduce Non-Linearity. And ReLU does a great job in introducing the same. Three reasons I choose ReLU as an Activation Function First it'...
user avatar
4 votes

Alternatives to linear activation function in regression tasks to limit the output

Don't use that activation function shown in the question. It doesn't do what you think it does. Instead: If you want the output to be in the range $[-1,1]$, you can use a sigmoid or tanh activation ...
user avatar
  • 2,968
4 votes

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

Just to add another solution. You can also write something like import tensorflow as tf keras = tf.keras ...
user avatar
4 votes
Accepted

Can we use ReLU activation function as the output layer's non-linearity?

Yes, you can. Basically, for regression tasks, it is customary to use the linear function as the non-linearity due to the fact that it's differentiable and it does not limit the output. This means you ...
user avatar
  • 13.3k
4 votes

How batch normalization layer resolve the vanishing gradient problem?

Batch Normalization (BN) does not prevent the vanishing or exploding gradient problem in a sense that these are impossible. Rather it reduces the probability for these to occur. Accordingly, the ...
user avatar
  • 4,807
3 votes
Accepted

Properly using activation functions of neural network

How is data predicted from activation functions? (considering that it returns constants on weighted input sum). You should consider the fact that the label of your input data is going to be ...
user avatar
  • 13.3k
3 votes

Is classifier able to say there's no-such-case?

Many classifiers do not directly give you L or R. They will give you the option which has a higher decision metric. For Naive Bayes this would be the class with the higher probability between $p(L|x)$ ...
user avatar
  • 8,478
3 votes

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

I don't know of any papers about this topic, but intuitively it makes a lot of sense to use monotonic activation functions. Let's say we have a non-monotonic activation function, maybe a Gaussian ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible