People who code: we want your input. Take the Survey

Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
0
votes
1answer
50 views

Example of convex activation function

For a particular task, I need a convex activation function with the following properties: f''(x) > 0 0 <= f(x) <= 1 f(x) is monotonic f(x) is not "exploding" i.e. avoiding functions such as ...
0
votes
0answers
22 views

Can this dataset be separated linearly?

Is this dataset linearly separable? If not, can it be converted into one by applying some function as it seems to follow the same pattern? Also, which classification algorithms could be used to fit ...
0
votes
1answer
29 views

Applying activation on part of the layer in Keras

Context I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network: ...
2
votes
1answer
124 views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
1
vote
1answer
78 views

Should output data scaling correspond to the activation function's output?

I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data ...
0
votes
1answer
19 views

Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
0
votes
1answer
32 views

Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...
1
vote
1answer
21 views

Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
3
votes
4answers
79 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
1answer
30 views

Training set Distribution and Activation function/Loss function correlation

How should the probability distribution of the training set influence the choice of the activation function / loss function? For instance if I have a Multinoulli distribution, which activation ...
7
votes
2answers
829 views

Why leaky relu is not so common in real practice?

As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have. Yet Leaky relu is less popular than Relu in real practice. Can someone tell ...
2
votes
1answer
23 views

Do Activation Functions map to Higher Dimensions?:

I just started learning tensorflow and I have a question regarding activation functions used in neural networks, I watched a 3b1b video a while ago and it seems it squished the value into an interval ...
0
votes
1answer
30 views

Numerically stable hyperbolic tangent

The hyperbolic tangent is commonly used as an activation function: $$ tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} $$ Although, it is unclear how this function is implemented to be numerically stable ...
1
vote
1answer
199 views

Relationship between Sigmoid and Gaussing Distribution

I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?": The assumption of a dependent variable to follow ...
2
votes
2answers
862 views

Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
0
votes
1answer
23 views

Input and output layer activation functions of neurons in Orange

What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?
6
votes
3answers
7k views

Activation function vs Squashing function

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer. Today, in a video explaining deep neural networks, I came across the term Squashing ...
0
votes
1answer
15 views

Can there be in-active neuron in output layer

I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a ...
35
votes
3answers
21k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
29
votes
2answers
19k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + ...
1
vote
3answers
703 views

What are the best activation functions for Binary text classification in neural networks?

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification. I have heard that Relu is best for Binary ...
1
vote
0answers
16 views

Does the number of hidden layers affect the activation function?

Suppose there's a network with N hidden layers. There are 2 cases: The network is deep The network is shallow I've been wondering how ...
0
votes
0answers
13 views

Guidelines to efficiently train neural networks which have polynomial activations

I am interesting in studying low footprint NN by replacing the activation functions with low-degree polynomial approximations. Doing this, I am fine with slight reduction in accuracy if I the training ...
3
votes
1answer
80 views

Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is used as an activation function that serves two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem, we use Gradient Clipping ...
0
votes
0answers
28 views

tanh function values are either 1 or -1, how to interpret that distribution

I have a question regarding the tanh function. I trained an NN (with tanh activation functions in hidden layers) on a multiclass dataset and visualised the tanh values of the complete samples from the ...
0
votes
1answer
33 views

Deep Neural Network and Activation Function

I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in, Increasing model complexity and introduce non-linearity Avoiding Gradient Vanishing ...
0
votes
0answers
13 views

Sigmoid vs Softmax with cross entropy

For a binary classifier, Is there any difference in Sigmoid activation function vs Softmax with cross entropy ? In this answer how-to-maximize-recall it is stated that if we use softmax with cross ...
23
votes
4answers
35k views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
1
vote
1answer
65 views

Is it possible to get an ROC curve using Relu activation?

Based on my understanding, given that Relu doesn't provide probabilities unlike Softmax, it's not possible to plot an ROC curve. However, is there some way to convert the output from a Relu to ...
4
votes
1answer
922 views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
0
votes
0answers
11 views

Sigmoid activation function for scaled continuous data

I've been working on a NLP project that attempt to output a single numeric value. The natural form of the data is integers between 0 and 27, with 27 being an absolute maximum, and values above 27 ...
0
votes
1answer
12 views

Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?

I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, ...
0
votes
0answers
9 views

How to determine activation functions for neural network

I am trying to plan a neural network for regression predictions. The final activation layer should be a linear function, but for hidden layers, do the activation functions need to also be all linear ...
1
vote
0answers
13 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
1
vote
1answer
38 views

Scaling the activation function

It is obvious that I have to scale the output data if the range of values is between say [-10;10] and the activation function of the output layer takes values in the interval [-1;1]. But I could also ...
0
votes
1answer
32 views

How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?

I was looking into the multi-label classification on the output layer of a Neural Network. I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons. So for example an ...
0
votes
3answers
2k views

Why there is no exact picture of softmax activation function?

I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
1
vote
1answer
161 views

Problem with convergence of ReLu in MLP

I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens. ...
0
votes
0answers
44 views

Avoid saturation in activation function (sigmoid)

Background It is said Sigmoid/Tanh are not to use because of the saturation issue. Question The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -...
1
vote
1answer
152 views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
1
vote
1answer
75 views

Dying leaky ReLU

I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
3
votes
2answers
484 views

What does the descision boundary of a relu look like?

A single non activated neuron is just a linear combination of its inputs. Thresholding this neuron's output as-is against 0 would create a hyperplane binary separator, whose parameters can be learned....
1
vote
0answers
219 views

What is the reason behind Keras choice of default (recurrent) activation functions in LSTM networks

Activation function between LSTM layers In the above link, the answer to the question whether activation function are required for LSTM layers was answered as follows: as an LSTM unit already consists ...
0
votes
2answers
66 views

What is the meaning of colors in activation maps?

I am trying to understand how the neural network works. I plotted the intermediate layer activations and I got the following image. I used the matplotlib library and selected ...
3
votes
0answers
101 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
1
vote
1answer
185 views

Counting Number of Parameters in Neural Networks [closed]

Note: This is an academics based problem. So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of ...
2
votes
1answer
304 views

As RELU is not differentiable when it touches the x-axis, doesn't it effect training?

When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent. I am a beginner in deep ...
1
vote
1answer
162 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
2
votes
1answer
60 views

Output landscape of ReLU, Swish and Mish

I found the following figure in the original Mish paper (https://arxiv.org/abs/1908.08681). I understand that this figure describes how the loss is being changed, if the change is smooth or not. But ...
1
vote
0answers
21 views

Sharing parameters of an activation across layers of a neural network

Keras now provides advanced parametric activation layers like Leaky-ReLU PReLU. Each time I add this layer to a sequential model, an additional trainable parameter is added to graph. How can I make ...