43
In mathematics (linear algebra) a function is considered linear whenever a function$f: A \rightarrow B$ if for every $x$ and $y$ in the domain $A$ has the following property: $f(x) + f(y) = f(x+y)$. By definition the ReLU is $max(0,x)$. Therefore, if we split the domain from $(-\infty, 0]$ or $[0, \infty)$ then the function is linear. However, it's easy ...
34
GELU function
We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows:
$$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$
Note that this is a definition, not an equation (or a relation). Authors have provided some justifications for this proposal, e.g. a ...
24
The biggest advantage of ReLu is indeed non-saturation of its gradient, which greatly accelerates the convergence of stochastic gradient descent compared to the sigmoid / tanh functions (paper by Krizhevsky et al).
But it's not the only advantage. Here is a discussion of sparsity effects of ReLu activations and induced regularization. Another nice property ...
20
You can use the LeakyRelu layer, as in the python class, instead of just specifying the string name like in your example. It works similarly to a normal layer.
Import the LeakyReLU and instantiate a model
from keras.layers import LeakyReLU
model = Sequential()
# here change your line to leave out an activation
model.add(Dense(90))
# now add a ReLU ...
17
A similar question was asked on CV: Comprehensive list of activation functions in neural networks with pros/cons.
I copy below one of the answers:
One such a list, though not much exhaustive:
http://cs231n.github.io/neural-networks-1/
Commonly used activation functions
Every activation function (or non-linearity) takes a single number
and ...
12
I understand the advantages of ReLU, which is avoiding dead neurons during backpropagation.
This is not completely true. The neurons are not dead. If you use sigmoid-like activations, after some iterations the value of gradients saturate for most the neurons. The value of gradient will be so small and the process of learning happens so slowly. This is ...
10
To explain using the sample neural network you have provided:
Purpose of the multiple inputs: Each input represents a feature of the input dataset.
Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
Purpose of the output layer: Each neuron represents a given class of the output (...
10
Sigmoid function is a partial case of softmax, when the number of classes $K=2$. That's why the similarity of their derivatives shouldn't surprise you.
Why do so many functions used in data science have derivatives of the
form f(x)*(1-f(x))?
If you consider the following differential equation
$y' = y \cdot (1-y)$
you will find the general solution in ...
10
First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show that $$\mathrm{erf}\left(\frac x {\sqrt2}\right) \approx \tanh\left(\sqrt{\frac2\pi} \left(x + a x^3\right)\right)$$ for $a \approx 0.044715$.
For large values ...
9
You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero, because each derivative could be potentially larger than one (up to $n$).
However, for practical purposes, we should ask ourselves how easy it is to maintain this situation (keeping the multiplication of derivatives away from zero)? Which turns out to be ...
9
First you need to define a function using backend functions. As an example, here is how I implemented the swish activation function:
from keras import backend as K
def swish(x, beta=1.0):
return x * K.sigmoid(beta * x)
This allows you to add the activation function to your model like this:
model.add(Conv2D(64, (3, 3)))
model.add(Activation(swish))
...
8
If your task is a kind of classification that the labels are mutually exclusive, each input just has one label, you have to use Softmax. If the inputs of your classification task have multiple labels for an input, your classes are not mutually exclusive and you can use Sigmoid for each output. For the former case, you should choose the output entry with the ...
7
Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and where the neuron of interest is. In principle the activation function $f$ would go to the calculation of the outcome
$$ y = f(Wx + b)$$
For output neurons, if ...
6
You have to normalize your data to accelerate learning process but based on experience its better to normalize your data in the standard manner, mean zero and standard deviation one. Although mapping to other small intervals near to zero may also be fine but the latter case usually takes more time than the other. If you use ReLU, again based on experience, ...
5
Activation functions like sigmoid function, hyperbolic tangent function, etc. are also called squashing function because they squash the input into a small range like in sigmoid function output is in range of [-1,1]. But you cannot call ReLU as a squashing function because for a positive input value it returns the output as same.
4
In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)
In certain scenarios having an activation with mean 0 is important (so tanh makes sense).
Sigmoid activation in the output layer is still important for classification
In 95% of the cases ReLU is much better though.
4
The addition of the activation layer creates a composition of two functions.
"A general function, to be defined for a particular context, is usually denoted by a single letter, most often the lower-case letters f, g, h."
So it comes down to the reason that he uses the hypothesis representation h(x)=wX+b which is a function, and that is wrapped by an ...
4
Taking your questions one after the other:
Assume that I have a classifier which can classify left hand / right
hand well. I am curious whether it can decide whether there's a hand
in the image?
No. It cannot. A model which is trained on hand/no-hand can do that. Not this. At most, it shall predict one of the either (left/hand) with a very low ...
4
The main reason to use an Activation Function in NN is to introduce Non-Linearity. And ReLU does a great job in introducing the same.
Three reasons I choose ReLU as an Activation Function
First it's Non-Linear( although it's acts like a linear function for x > 0)
ReLU is cheap to compute. Since it's simple math, model takes less time to run
ReLU induces ...
4
Don't use that activation function shown in the question. It doesn't do what you think it does. Instead:
If you want the output to be in the range $[-1,1]$, you can use a sigmoid or tanh activation function in the final layer.
If you want the output to be in the range $[-100,100]$, do the same, then multiply the final output by 100 (a hardcoded constant)....
4
softmax() will give you the probability distribution which means all output will sum to 1. While, sigmoid() will make sure the output value of neuron is between 0 to 1.
In case of digit classification and sigmoid(), you will have output of 10 output neurons between 0 to 1. Then, you can take biggest one of them and classify as that digit.
4
Yes, you can. Basically, for regression tasks, it is customary to use the linear function as the non-linearity due to the fact that it's differentiable and it does not limit the output. This means you can make any output using your inputs. People do not use tanh or sigmoid as the activation function of the last layers for regression tasks due to the fact ...
4
Batch Normalization (BN) does not prevent the vanishing or exploding gradient problem in a sense that these are impossible. Rather it reduces the probability for these to occur. Accordingly, the original paper states:
In traditional deep networks, too-high learning rate may result in the gradients that explode or vanish, as well as getting stuck in poor ...
3
How is data predicted from activation functions? (considering that it returns constants on weighted input sum).
You should consider the fact that the label of your input data is going to be predicted by the network. Moreover, the outputs of the network usually represent the probability of belonging to each class. The label is not predicted by the ...
3
An activation function
This the name given to a function, which is applied to a neuron that just had a weight update as a result of new information. It can refer to any of the well known activation funtions, such as the Rectified Linear Unit (ReLU), the hyperbolic tangent function (tanh) or even the identity function! Have a look at somewhere like the Keras ...
3
Many classifiers do not directly give you L or R. They will give you the option which has a higher decision metric. For Naive Bayes this would be the class with the higher probability between $p(L|x)$ and $p(R|x)$. This is true for most classifiers.
What you can do is set a hyper-parameter (if you have some anomalous instances) to determine how well ...
3
I don't know of any papers about this topic, but intuitively it makes a lot of sense to use monotonic activation functions. Let's say we have a non-monotonic activation function, maybe a Gaussian kernel, symmetric around $x=0$ but slides off towards $f(x)=0$ if x strays away from 0 on either side. If we have a sample that we feed into our network that ...
3
This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.
Generally you want a linear output, unless you can guarantee scaling total possible reward to within a limited range such as $[-1,1]$ for $\text{tanh}$. Reminder this is not for estimating individual ...
3
Sigmoid helps in controlling the activation unlike ReLu which blows up it up. Sigmoids don't overfit as much.
Have a look at this.
I still would ask you to start with ReLu for training as it performs better most of the time.
3
I believe the question was about using LeayReLU within the Keras Functional API. Which would look something like this:
from keras.layers import LeakyReLU
...
x = Dense(128)(x)
x = LeakyReLU(alpha=0.3)(x)
```
Only top voted, non community-wiki answers of a minimum length are eligible
