Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
0
votes
0answers
21 views

What will be the activation functions of $g_1, g_2, g_3, g_4$?

I recently had an interview question that posed the following The forward propagation equation for a neuron with three inputs is given by: where g_i is the activation function for the neuron. Assume ...
1
vote
0answers
16 views

Does the number of hidden layers affect the activation function?

Suppose there's a network with N hidden layers. There are 2 cases: The network is deep The network is shallow I've been wondering how ...
0
votes
0answers
12 views

Guidelines to efficiently train neural networks which have polynomial activations

I am interesting in studying low footprint NN by replacing the activation functions with low-degree polynomial approximations. Doing this, I am fine with slight reduction in accuracy if I the training ...
0
votes
0answers
27 views

tanh function values are either 1 or -1, how to interpret that distribution

I have a question regarding the tanh function. I trained an NN (with tanh activation functions in hidden layers) on a multiclass dataset and visualised the tanh values of the complete samples from the ...
0
votes
0answers
8 views

Sigmoid vs Softmax with cross entropy

For a binary classifier, Is there any difference in Sigmoid activation function vs Softmax with cross entropy ? In this answer how-to-maximize-recall it is stated that if we use softmax with cross ...
0
votes
0answers
7 views

Sigmoid activation function for scaled continuous data

I've been working on a NLP project that attempt to output a single numeric value. The natural form of the data is integers between 0 and 27, with 27 being an absolute maximum, and values above 27 ...
1
vote
0answers
14 views

Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
0
votes
1answer
10 views

Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?

I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, ...
0
votes
0answers
9 views

How to determine activation functions for neural network

I am trying to plan a neural network for regression predictions. The final activation layer should be a linear function, but for hidden layers, do the activation functions need to also be all linear ...
1
vote
0answers
13 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
0
votes
1answer
29 views

How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?

I was looking into the multi-label classification on the output layer of a Neural Network. I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons. So for example an ...
1
vote
1answer
21 views

Scaling the activation function

It is obvious that I have to scale the output data if the range of values is between say [-10;10] and the activation function of the output layer takes values in the interval [-1;1]. But I could also ...
1
vote
1answer
112 views

Problem with convergence of ReLu in MLP

I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens. ...
0
votes
0answers
36 views

Avoid saturation in activation function (sigmoid)

Background It is said Sigmoid/Tanh are not to use because of the saturation issue. Question The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -...
1
vote
1answer
60 views

Dying leaky ReLU

I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
1
vote
1answer
59 views

Is it possible to get an ROC curve using Relu activation?

Based on my understanding, given that Relu doesn't provide probabilities unlike Softmax, it's not possible to plot an ROC curve. However, is there some way to convert the output from a Relu to ...
1
vote
0answers
149 views

What is the reason behind Keras choice of default (recurrent) activation functions in LSTM networks

Activation function between LSTM layers In the above link, the answer to the question whether activation function are required for LSTM layers was answered as follows: as an LSTM unit already consists ...
3
votes
0answers
85 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
2
votes
1answer
214 views

As RELU is not differentiable when it touches the x-axis, doesn't it effect training?

When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent. I am a beginner in deep ...
2
votes
1answer
53 views

Output landscape of ReLU, Swish and Mish

I found the following figure in the original Mish paper (https://arxiv.org/abs/1908.08681). I understand that this figure describes how the loss is being changed, if the change is smooth or not. But ...
1
vote
0answers
19 views

Sharing parameters of an activation across layers of a neural network

Keras now provides advanced parametric activation layers like Leaky-ReLU PReLU. Each time I add this layer to a sequential model, an additional trainable parameter is added to graph. How can I make ...
0
votes
0answers
28 views

Neural network output is 0 for test data (using RELU for activation)

Maybe this is a naive question, but I have a NN that uses relu for all layers. In train data there is no problem, but in test (or validation) the outputs are all 0. ...
1
vote
0answers
19 views

Using Iterative Hard/Soft Thresholding in autoencoder with non linear activation

Can someone please give an intuitive explanation of the difference between the Iterative Hard Thresholding VS Iterative Soft thresholding algorithm? And if we can use these algorithms in an ...
0
votes
0answers
18 views

Conv1D relu: negative values or additional sequence?

If I have sequences that can have negative values with occasional spikes in both positive and negative directions I'd like to preserve, what is the best practice to handle those sequences: should I ...
0
votes
0answers
20 views

proper activation function at output and loss function to optimize for OCR?

I am trying to make a CNN model on IAM handwritten words data(which has images of words handwritten by multiple people and targets are text in the images). So, I can encode words to numbers(A=0, B=1 ...
1
vote
1answer
98 views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
1
vote
0answers
13 views

Wich activation function for DQL

After many research, I still can't find a neat answer about this question: When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
2
votes
2answers
390 views

What does the descision boundary of a relu look like?

A single non activated neuron is just a linear combination of its inputs. Thresholding this neuron's output as-is against 0 would create a hyperplane binary separator, whose parameters can be learned....
7
votes
1answer
664 views

Why leaky relu is not so common in real practice?

As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have. Yet Leaky relu is less popular than Relu in real practice. Can someone tell ...
0
votes
2answers
51 views

What is the meaning of colors in activation maps?

I am trying to understand how the neural network works. I plotted the intermediate layer activations and I got the following image. I used the matplotlib library and selected ...
1
vote
1answer
641 views

Setting activation function to a leaky relu in a Sequential model

I'm doing a beginner's TensorFlow course, we are given a mini-project about predicting the MNIST data set (hand written digits) and we have to finish the code such that we get a 99% accuracy (measured ...
0
votes
1answer
32 views

In backpropagation, scale is also important?

I think backpropagation is needed to find the direction of gradient decent method. I also wonder, the scale is also important? I heard some issue of vanishing(or exploding) gradient problem. If the ...
0
votes
1answer
43 views

Example of convex activation function

For a particular task, I need a convex activation function with the following properties: f''(x) > 0 0 <= f(x) <= 1 f(x) is monotonic f(x) is not "exploding" i.e. avoiding functions such as ...
0
votes
2answers
46 views

Is there a limit in the number of layers for neural network?

I heard the neural network has a problem with vanishing gradient problems even though the ReLU activation function is used. In ResNet(that has a connecting function for reducing the problem), there ...
1
vote
1answer
175 views

What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE). So we have $$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$ Now we have BCE: $$ ...
1
vote
1answer
15 views

Relu with not gradient vanishing function is possible?

I'm beginner in ML. In the ANN, relu has the gradient of 1 in x>0 how ever, i wonder in x=<0 relu has gradient of 0 and may have gradient vanishing problem in deep neural networks. if activation ...
1
vote
1answer
62 views

Should output data scaling correspond to the activation function's output?

I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data ...
0
votes
0answers
20 views

If input data to CNN is not normalised, how should I initialise the weights?

I've read that He normalisation is preferred for Relu activated CNN's. However, understanding how Relu's work by linearly activating positive inputs while zero or negative inputs are zero'd (with no ...
2
votes
1answer
747 views

Different activation function in same layer of a Neural network

My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers. Suppose I have 3 ...
1
vote
0answers
151 views

Generalized softmax derivative for implementation with any loss function

I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
1
vote
0answers
26 views

Which activation function of the output layer and which loss function are advised to be used for bounded regression?

I want my (deep) neural network to produce an output from a certain range, in my case between 0 and 255. I have scaled the labels from [0..255] to [0..1]. For the neural network, I have tried a ...
1
vote
0answers
178 views

Writing a piecewise linear function as a sum of ReLU functions

Suppose I have a piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, where $\{\phi_i\}_{i=1}^n$ is a finite dimension space of dimension $n-1$, in particular I am interested in the functions ...
0
votes
0answers
82 views

Is it reasonable to use the output of the sigmoid function as the win rate prediction?

I'm working on a project which is predicting the win rate of one team or one person. (could be any kind of sports like baseball, basketball or e-sport games) The data I have is more like a ...
3
votes
2answers
141 views

What came first? Backpropagation or Sigmoid?

Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function. When did the sigmoid function become so popular in NNs?
3
votes
1answer
73 views

Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is used as an activation function that serves two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem, we use Gradient Clipping ...
4
votes
6answers
899 views

Why is activation needed at all in neural network?

I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know ...
3
votes
1answer
1k views

Leaky ReLU inside of a Simple Python Neural Net

To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better ...
1
vote
0answers
16 views

How does one use activation function with greater than [-1;1] range for binary classification?

In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes. They explain (§ 4....
3
votes
0answers
131 views

How does one derive the modified tanh activation proposed by LeCun?

In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form: $$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$ ...
4
votes
1answer
769 views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...