Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
6
votes
1answer
83 views

Why leaky relu is not so common in real practice?

As leaky relu does not lead any value to 0, so training always continues. And I can't think of any disadvantages it have. Yet Leaky relu is less popular than Relu in real practice. Can someone tell ...
0
votes
1answer
12 views

What is the meaning of colors in activation maps?

I am trying to understand how the neural network works. I plotted the intermediate layer activations and I got the following image. I used the matplotlib library and selected ...
1
vote
1answer
21 views

Setting activation function to a leaky relu in a Sequential model

I'm doing a beginner's TensorFlow course, we are given a mini-project about predicting the MNIST data set (hand written digits) and we have to finish the code such that we get a 99% accuracy (measured ...
0
votes
1answer
14 views

In backpropagation, scale is also important?

I think backpropagation is needed to find the direction of gradient decent method. I also wonder, the scale is also important? I heard some issue of vanishing(or exploding) gradient problem. If the ...
0
votes
1answer
11 views

Example of convex activation function

For a particular task, I need a convex activation function with the following properties: f''(x) > 0 0 <= f(x) <= 1 f(x) is monotonic f(x) is not "exploding" i.e. avoiding functions such as ...
0
votes
2answers
26 views

Is there a limit in the number of layers for neural network?

I heard the neural network has a problem with vanishing gradient problems even though the ReLU activation function is used. In ResNet(that has a connecting function for reducing the problem), there ...
1
vote
1answer
12 views

What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE). So we have $$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$ Now we have BCE: $$ ...
1
vote
1answer
4 views

Relu with not gradient vanishing function is possible?

I'm beginner in ML. In the ANN, relu has the gradient of 1 in x>0 how ever, i wonder in x=<0 relu has gradient of 0 and may have gradient vanishing problem in deep neural networks. if activation ...
1
vote
1answer
9 views

Should output data scaling correspond to the activation function's output?

I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data ...
0
votes
0answers
16 views

If input data to CNN is not normalised, how should I initialise the weights?

I've read that He normalisation is preferred for Relu activated CNN's. However, understanding how Relu's work by linearly activating positive inputs while zero or negative inputs are zero'd (with no ...
2
votes
1answer
68 views

Different activation function in same layer of a Neural network

My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers. Suppose I have 3 ...
1
vote
0answers
62 views

Generalized softmax derivative for implementation with any loss function

I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
1
vote
0answers
17 views

Which activation function of the output layer and which loss function are advised to be used for bounded regression?

I want my (deep) neural network to produce an output from a certain range, in my case between 0 and 255. I have scaled the labels from [0..255] to [0..1]. For the neural network, I have tried a ...
0
votes
0answers
39 views

Writing a piecewise linear function as a sum of ReLU functions

Suppose I have a piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, where $\{\phi_i\}_{i=1}^n$ is a finite dimension space of dimension $n-1$, in particular I am interested in the functions ...
0
votes
0answers
45 views

Is it reasonable to use the output of the sigmoid function as the win rate prediction?

I'm working on a project which is predicting the win rate of one team or one person. (could be any kind of sports like baseball, basketball or e-sport games) The data I have is more like a ...
3
votes
2answers
50 views

What came first? Backpropagation or Sigmoid?

Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function. When did the sigmoid function become so popular in NNs?
3
votes
1answer
37 views

Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is used as a an activation function which serve two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem we use Gradient Clipping ...
4
votes
6answers
883 views

Why is activation needed at all in neural network?

I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know ...
3
votes
1answer
202 views

Leaky ReLU inside of a Simple Python Neural Net

To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better ...
1
vote
0answers
7 views

How does one use activation function with greater than [-1;1] range for binary classification?

In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes. They explain (§ 4....
3
votes
0answers
44 views

How does one derive the modified tanh activation proposed by LeCun?

In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form: $$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$ ...
3
votes
1answer
128 views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
0
votes
1answer
46 views

PELU activation: how does it work, and how to implement?

I have encountered PELU (Parametric Exponential Linear Unit) in the literature, but I can't find practical applications of it. Moreover, I have some questions about how it works: Are its parameters ...
3
votes
1answer
25 views

How to quantitatively evaluate raw neural network activations?

Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam ...
1
vote
1answer
54 views

Activation function vs If else statement

The question is very naive and most of us may know the answer. I have googled it but was not able to find a satisfactory answer so posting it here. Can someone please put the right words on this ...
3
votes
2answers
353 views

TensorFlow Sigmoid activation function as output layer - value interpretation

My TensorFlow model has the following structure. It aims to solve a binary classification problem where the labels are either 0 or 1. The output layer uses a sigmoid activation function with 1 output. ...
1
vote
1answer
114 views

Binary classifier using Keras with backend Tensorflow with a Binary output

I am trying to build a binary classifier with tensorflow.keras Currently unable to identify a solution to having the model generating only 0s and 1s. The code for compiling my tensorflow model. <...
2
votes
1answer
58 views

Counting Number of Parameters in Neural Networks

Note: This is an academics based problem. So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of ...
1
vote
0answers
16 views

Should I scale/normalize the data before training a feedforward neural network using only lagged values?

So I am trying to forecast the price of a certain commodity that is quite volatile. Now I wanted to train a regression feedforward neural network to predict the price of next week. Just to be clear, ...
1
vote
1answer
53 views

Relationship between Sigmoid and Gaussing Distribution

I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?": The assumption of a dependent variable to follow ...
-1
votes
2answers
51 views

Activation Functions in Neural network

I have a set of questions related to the usage of various activation functions used in neural networks. I would highly appreciate if someone could give explanatory answers. Why is ReLU is used only ...
3
votes
2answers
75 views

Why activation functions used in neural networks generally have limited range?

Why do we generally use activation functions with only limited range in neural networks? for e.g. $sigmoid$ activation function has range $[0, 1]$ $tanh$ activation function has range $[-1, 1]$ Q1) ...
0
votes
0answers
13 views
1
vote
1answer
49 views

Square-law based RBF kernel

What is the Square-law based RBF kernel (SQ-RBF)? The definition in the table at the Wikipedia article Activation Function looks wrong, since it says ...
1
vote
1answer
90 views

Combining multiple neural networks with different activation functions

I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the ...
1
vote
2answers
477 views

ReLU for combating the problem of vanishing gradient in RNN?

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used. When we talk about solving the vanishing gradient problem in RNN, we use a more ...
0
votes
1answer
81 views

Modulo as activation function in neural network

Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our ...
7
votes
3answers
4k views

How to create custom Activation functions in Keras / TensorFlow?

I'm using keras and I wanted to add my own activation function myf to tensorflow backend. how to define the new function and make it operational. so instead of the line of code: ...
0
votes
0answers
53 views

Effect of ReLu derivative in convolution layer backpropagation

I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images. Following this ...
0
votes
0answers
49 views

How exactly should i use softmax activation in last layer of a Neural Network

I am developing a digit classifer with MNIST Dataset. I have read that for classification problems softmax activation function is used, as it maps last layer neuron's outputs into probabilities. ...
0
votes
3answers
439 views

Why there is no exact picture of softmax activation function?

I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
0
votes
1answer
208 views

What are the best activation functions for Binary text classification in neural networks?

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification. I have heard that Relu is best for Binary ...
0
votes
0answers
18 views

Piecewise-linear activation function definition doesn't correspond to graph found in textbook

This question is a duplicate of this question on StackOverflow, but it hasn't been answered in 2 years, which is why I'm re-posting it. If this isn't an appropriate way to go about this, please let me ...
0
votes
0answers
13 views

What are contexts where a perceptron model could be defined?

per this post, a perceptron could use the logit function as the activation function. per wiki In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step ...
0
votes
1answer
64 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
0
votes
0answers
81 views

Thresholding in intermediate layer using Gumbel Softmax

In a neural network, for an intermediate layer, I need to threshold the output. The output of each neuron in the layer is a real value, but I need to binarize it (to 0 or 1). But with hard ...
0
votes
1answer
22 views

Deep Neural Network and Activation Function

I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in, Increasing model complexity and introduce non-linearity Avoiding Gradient Vanishing ...
0
votes
3answers
162 views

How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for ...
2
votes
1answer
3k views

Using LeakyRelu as activation function in CNN and best alpha for it

Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write: <...
1
vote
1answer
53 views

Confusion regarding the Working mechanism of Activation function

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...