All Questions
Tagged with neural-network activation-function
91 questions
62
votes
3
answers
62k
views
LeakyReLU vs PReLU
I thought both, PReLU and Leaky ReLU are:
$$f(x) = \max(x, \alpha x) \qquad \text{ with } \alpha \in (0, 1)$$
Keras, however, has both functions in the docs.
Leaky ReLU
Source of LeakyReLU:
...
- 19.2k
48
votes
4
answers
35k
views
Why is ReLU used as an activation function?
Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.
Which I am able to understand ...
- 603
23
votes
2
answers
39k
views
Why ReLU is better than the other activation functions
Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
- 14.2k
22
votes
1
answer
20k
views
Difference of Activation Functions in Neural Networks in general
I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear.
It's reasonable that one ...
- 8,697
19
votes
2
answers
12k
views
Why deep learning models still use RELU instead of SELU, as their activation function?
I am a trying to understand the SELU activation function and I was wondering why deep learning practitioners keep using RELU, with all its issues, instead of SELU, which enables a neural network to ...
14
votes
1
answer
16k
views
Input normalization for ReLu?
Let's assume a vanilla MLP for classification with a given activation function for hidden layers.
I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
- 243
13
votes
5
answers
37k
views
How does Sigmoid activation work in multi-class classification problems
I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
- 131
11
votes
3
answers
5k
views
Why Leaky ReLU is not so common in real practice?
As Leaky ReLU does not lead any value to 0, so training always continues. And I can't think of any disadvantages it has.
Yet Leaky ReLU is less popular than ReLU in real practice. Can someone tell why ...
- 221
10
votes
4
answers
12k
views
Activation function vs Squashing function
This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer.
Today, in a video explaining deep neural networks, I came across the term Squashing ...
- 203
10
votes
1
answer
6k
views
Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?
ReLU is an activation function defined as $h = \max(0, a)$ where $a = Wx + b$.
Normally, we train neural networks with first-order methods such as SGD, Adam, RMSprop, Adadelta, or Adagrad. ...
- 2,216
10
votes
2
answers
25k
views
ReLU vs Leaky ReLU vs ELU with pros and cons
I am unable to understand when to use ReLU, Leaky ReLU and ELU.
How do they compare to other activation functions(like the sigmoid and the tanh) and their pros and cons.
- 113
8
votes
4
answers
4k
views
Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?
To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
- 131
8
votes
2
answers
21k
views
What activation function should I use for a specific regression problem?
Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain ...
- 295
8
votes
2
answers
612
views
How does one derive the modified tanh activation proposed by LeCun?
In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form:
$$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$
...
- 2,444
7
votes
3
answers
5k
views
What is the purpose of multiple neurons in a hidden layer?
On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.
Let me make the question more clear.
Take this ...
- 477
7
votes
2
answers
12k
views
Gradient Descent in ReLU Neural Network
I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
- 71
7
votes
1
answer
3k
views
Is it wrong to use Glorot Initialization with ReLu Activation?
I'm reading that keras' default initialization is glorot_uniform.
However, all of the tutorials I see are using relu ...
- 539
6
votes
2
answers
1k
views
Advantages of monotonic activation functions over non-monotonic functions in neural networks?
What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks?
Do they perform better than non-monotonic ones?
Is this mathematically proven?
Are ...
- 403
5
votes
1
answer
283
views
Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?
The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
- 161
5
votes
2
answers
622
views
What came first? Backpropagation or Sigmoid?
Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function.
When did the sigmoid function become so popular in NNs?
- 419
5
votes
0
answers
189
views
Intuitively, why do Non-monotonic Activations Work?
The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
- 73
4
votes
3
answers
2k
views
What does the descision boundary of a relu look like?
A single non activated neuron is just a linear combination of its inputs.
Thresholding this neuron's output as-is against 0 would create a hyperplane binary separator, whose parameters can be learned....
- 196
4
votes
1
answer
1k
views
Is there a way to set a different activation function for each hidden unit in one layer in keras?
I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
- 403
3
votes
4
answers
1k
views
Neural Network not Deep
I have found this image link I would like to know what NNs are not deep neural? The first three?
Also what kind of functional activations do they use?
- 491
3
votes
3
answers
5k
views
Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?
Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:
simple
more biologically plausible
no gradient to vanish
better performance
sparsity
And I see only one ...
- 849
3
votes
2
answers
162
views
Is classifier able to say there's no-such-case?
I am a starter in ML and I need some help...
The problem
Assume that I have a classifier which can classify left hand / right hand well.
I am curious whether it can decide whether there's a hand in ...
- 31
3
votes
4
answers
12k
views
Why there is no exact picture of softmax activation function?
I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
- 432
3
votes
1
answer
5k
views
Can we use ReLU activation function as the output layer's non-linearity?
I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value.
Can I use ...
- 463
3
votes
1
answer
1k
views
Leaky ReLU inside of a Simple Python Neural Net
To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better ...
- 133
3
votes
2
answers
775
views
Why activation functions used in neural networks generally have limited range?
Why do we generally use activation functions with only limited range in neural networks? for e.g.
$sigmoid$ activation function has range $[0, 1]$
$tanh$ activation function has range $[-1, 1]$
Q1) ...
- 175
3
votes
1
answer
877
views
Why the sigmoid activation function results in sub-optimal gradient descent?
I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
- 33
3
votes
1
answer
38
views
How to quantitatively evaluate raw neural network activations?
Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam ...
- 539
3
votes
1
answer
200
views
Product of dot products in neural network
In a neural network, it is common to compute a dot product of the form
$$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$
and use it as argument to some activation function. This is ...
- 133
3
votes
4
answers
152
views
Few activation functions handling various problems - neural networks
How can a few activation functions in neural networks handle so many different problems?
I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
- 161
2
votes
1
answer
624
views
How to prove Softmax Numerical Stability?
I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of ...
- 21
2
votes
2
answers
3k
views
Alternatives to linear activation function in regression tasks to limit the output
I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
- 14.2k
2
votes
1
answer
2k
views
As RELU is not differentiable when it touches the x-axis, doesn't it effect training?
When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent.
I am a beginner in deep ...
- 709
2
votes
1
answer
3k
views
Different activation function in same layer of a Neural network
My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers.
Suppose I have 3 ...
- 143
2
votes
1
answer
2k
views
Implementing a custom hard sigmoid function
I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values:
0 if x < 0
1 if x > 1
x if 0 <= x <= 1
How do I implement it with a ...
- 2,250
2
votes
1
answer
63
views
Question about non linearity of activation function
I have a basic question about activation functions. It is told that they are added to the network to introduce non linearity.
However, the neural network itself is non linear. Isn' it? If we see any ...
- 914
2
votes
2
answers
429
views
Dying leaky ReLU
I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
2
votes
3
answers
4k
views
What are the best activation functions for Binary text classification in neural networks?
I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification.
I have heard that Relu is best for Binary ...
2
votes
0
answers
103
views
Derive backpropagation for PreLU
I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows:
$$
h_a(x) = \text{max}(ax, x)
$$
I want to derive $ \frac{\partial L}{\partial ...
- 21
2
votes
1
answer
2k
views
Relationship between Sigmoid and Gaussing Distribution
I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?":
The assumption of a dependent variable to follow ...
- 21
2
votes
0
answers
124
views
Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?
My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus.
My main purpose is to ...
2
votes
0
answers
183
views
Why isn't Maxout used in the state of the art models?
I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks".
It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
- 121
1
vote
2
answers
135
views
Weights in neural network
So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data ...
- 149
1
vote
1
answer
962
views
Whats the advantage of He Intialization over Xavier Intialization?
For Weights initialization, I read that He doesn't consider linear activation of neurons as Xavier Initialization; in this context, what does linear initialization mean?
- 13
1
vote
1
answer
247
views
Example of convex activation function
For a particular task, I need a convex activation function with the following properties:
f''(x) > 0
0 <= f(x) <= 1
f(x) is monotonic
f(x) is not "exploding" i.e. avoiding functions such as ...
- 111
1
vote
1
answer
160
views
Confusion regarding the Working mechanism of Activation function
For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...
- 541