All Questions
34 questions
62
votes
3
answers
62k
views
LeakyReLU vs PReLU
I thought both, PReLU and Leaky ReLU are:
$$f(x) = \max(x, \alpha x) \qquad \text{ with } \alpha \in (0, 1)$$
Keras, however, has both functions in the docs.
Leaky ReLU
Source of LeakyReLU:
...
- 19.2k
48
votes
4
answers
35k
views
Why is ReLU used as an activation function?
Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network.
Which I am able to understand ...
- 603
23
votes
2
answers
39k
views
Why ReLU is better than the other activation functions
Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
- 14.2k
19
votes
2
answers
12k
views
Why deep learning models still use RELU instead of SELU, as their activation function?
I am a trying to understand the SELU activation function and I was wondering why deep learning practitioners keep using RELU, with all its issues, instead of SELU, which enables a neural network to ...
14
votes
1
answer
16k
views
Input normalization for ReLu?
Let's assume a vanilla MLP for classification with a given activation function for hidden layers.
I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
- 243
13
votes
5
answers
37k
views
How does Sigmoid activation work in multi-class classification problems
I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
- 131
11
votes
3
answers
5k
views
Why Leaky ReLU is not so common in real practice?
As Leaky ReLU does not lead any value to 0, so training always continues. And I can't think of any disadvantages it has.
Yet Leaky ReLU is less popular than ReLU in real practice. Can someone tell why ...
- 221
10
votes
4
answers
12k
views
Activation function vs Squashing function
This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer.
Today, in a video explaining deep neural networks, I came across the term Squashing ...
- 203
10
votes
2
answers
25k
views
ReLU vs Leaky ReLU vs ELU with pros and cons
I am unable to understand when to use ReLU, Leaky ReLU and ELU.
How do they compare to other activation functions(like the sigmoid and the tanh) and their pros and cons.
- 113
8
votes
4
answers
4k
views
Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?
To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
- 131
8
votes
2
answers
21k
views
What activation function should I use for a specific regression problem?
Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain ...
- 295
7
votes
3
answers
5k
views
What is the purpose of multiple neurons in a hidden layer?
On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.
Let me make the question more clear.
Take this ...
- 477
6
votes
2
answers
1k
views
Advantages of monotonic activation functions over non-monotonic functions in neural networks?
What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks?
Do they perform better than non-monotonic ones?
Is this mathematically proven?
Are ...
- 403
5
votes
1
answer
283
views
Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?
The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
- 161
5
votes
0
answers
189
views
Intuitively, why do Non-monotonic Activations Work?
The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
- 73
4
votes
1
answer
1k
views
Is there a way to set a different activation function for each hidden unit in one layer in keras?
I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
- 403
3
votes
2
answers
162
views
Is classifier able to say there's no-such-case?
I am a starter in ML and I need some help...
The problem
Assume that I have a classifier which can classify left hand / right hand well.
I am curious whether it can decide whether there's a hand in ...
- 31
3
votes
1
answer
5k
views
Can we use ReLU activation function as the output layer's non-linearity?
I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value.
Can I use ...
- 463
3
votes
2
answers
775
views
Why activation functions used in neural networks generally have limited range?
Why do we generally use activation functions with only limited range in neural networks? for e.g.
$sigmoid$ activation function has range $[0, 1]$
$tanh$ activation function has range $[-1, 1]$
Q1) ...
- 175
2
votes
2
answers
3k
views
Alternatives to linear activation function in regression tasks to limit the output
I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
- 14.2k
2
votes
1
answer
3k
views
Different activation function in same layer of a Neural network
My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers.
Suppose I have 3 ...
- 143
2
votes
2
answers
429
views
Dying leaky ReLU
I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
2
votes
0
answers
183
views
Why isn't Maxout used in the state of the art models?
I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks".
It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
- 121
1
vote
1
answer
962
views
Whats the advantage of He Intialization over Xavier Intialization?
For Weights initialization, I read that He doesn't consider linear activation of neurons as Xavier Initialization; in this context, what does linear initialization mean?
- 13
1
vote
1
answer
247
views
Example of convex activation function
For a particular task, I need a convex activation function with the following properties:
f''(x) > 0
0 <= f(x) <= 1
f(x) is monotonic
f(x) is not "exploding" i.e. avoiding functions such as ...
- 111
1
vote
1
answer
7k
views
Why is the softmax function often used as activation function of output layer in classification neural networks?
What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
- 1,445
1
vote
2
answers
240
views
Is there comprehensive list of activation functions and their applications for a Neural Network?
I am aware of common activation functions like Sigmoid, Tanh, ReLU, Leaky ReLU. Even heard about a function called Swish(SiLU). Now is there any detailed information on other activations functions and ...
- 11
1
vote
1
answer
1k
views
Problem with convergence of ReLu in MLP
I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
...
1
vote
0
answers
273
views
Generalized softmax derivative for implementation with any loss function
I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
- 11
0
votes
1
answer
628
views
Properly using activation functions of neural network
I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear.
Input Layer:
From my ...
- 409
0
votes
1
answer
369
views
What exactly is the "hyperbolic" tanh function used in the context of activation functions?
I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows
$$
\tanh(x) = \frac{\sinh(x)}{...
- 2,068
0
votes
0
answers
31
views
What happens if you don't include any activation function on hidden classification layers?
What happens if we don't apply an activation function to the classification hidden layers and apply it only for the final output layer (Sigmoid, Softmax)?
I'm asking this because I have trained a CNN ...
- 27
0
votes
1
answer
225
views
How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?
I was looking into the multi-label classification on the output layer of a Neural Network.
I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons.
So for example an ...
-1
votes
2
answers
87
views
Activation Functions in Neural network
I have a set of questions related to the usage of various activation functions used in neural networks. I would highly appreciate if someone could give explanatory answers.
Why is ReLU is used only ...
- 21