Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

135 questions
Filter by
Sorted by
Tagged with
98 views

Is it reasonable to use the output of the sigmoid function as the win rate prediction?

I'm working on a project which is predicting the win rate of one team or one person. (could be any kind of sports like baseball, basketball or e-sport games) The data I have is more like a ...
173 views

What came first? Backpropagation or Sigmoid?

Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function. When did the sigmoid function become so popular in NNs?
82 views

ReLU is used as an activation function that serves two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem, we use Gradient Clipping ...
905 views

Why is activation needed at all in neural network?

I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know ...
1k views

Leaky ReLU inside of a Simple Python Neural Net

To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better ...
18 views

How does one use activation function with greater than [-1;1] range for binary classification?

In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes. They explain (§ 4....
141 views

How does one derive the modified tanh activation proposed by LeCun?

In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form: $$f(x) = 1.7159 * tanh(\frac{2}{3}*x)$$ ...
935 views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
183 views

PELU activation: how does it work, and how to implement?

I have encountered PELU (Parametric Exponential Linear Unit) in the literature, but I can't find practical applications of it. Moreover, I have some questions about how it works: Are its parameters ...
26 views

How to quantitatively evaluate raw neural network activations?

Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam ...
153 views

Activation function vs If else statement

The question is very naive and most of us may know the answer. I have googled it but was not able to find a satisfactory answer so posting it here. Can someone please put the right words on this ...
2k views

TensorFlow Sigmoid activation function as output layer - value interpretation

My TensorFlow model has the following structure. It aims to solve a binary classification problem where the labels are either 0 or 1. The output layer uses a sigmoid activation function with 1 output. ...
417 views

Binary classifier using Keras with backend Tensorflow with a Binary output

I am trying to build a binary classifier with tensorflow.keras Currently unable to identify a solution to having the model generating only 0s and 1s. The code for compiling my tensorflow model. <...
187 views

Counting Number of Parameters in Neural Networks [closed]

Note: This is an academics based problem. So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of ...
58 views

Should I scale/normalize the data before training a feedforward neural network using only lagged values?

So I am trying to forecast the price of a certain commodity that is quite volatile. Now I wanted to train a regression feedforward neural network to predict the price of next week. Just to be clear, ...
205 views

Relationship between Sigmoid and Gaussing Distribution

I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?": The assumption of a dependent variable to follow ...
65 views

Activation Functions in Neural network

I have a set of questions related to the usage of various activation functions used in neural networks. I would highly appreciate if someone could give explanatory answers. Why is ReLU is used only ...
195 views

Why activation functions used in neural networks generally have limited range?

Why do we generally use activation functions with only limited range in neural networks? for e.g. $sigmoid$ activation function has range $[0, 1]$ $tanh$ activation function has range $[-1, 1]$ Q1) ...
85 views

Square-law based RBF kernel

What is the Square-law based RBF kernel (SQ-RBF)? The definition in the table at the Wikipedia article Activation Function looks wrong, since it says ...
190 views

Combining multiple neural networks with different activation functions

I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the ...
2k views

ReLU for combating the problem of vanishing gradient in RNN?

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used. When we talk about solving the vanishing gradient problem in RNN, we use a more ...
200 views

Modulo as activation function in neural network

Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our ...
11k views

How to create custom Activation functions in Keras / TensorFlow?

I'm using keras and I wanted to add my own activation function myf to tensorflow backend. how to define the new function and make it operational. so instead of the line of code: ...
2k views

Why there is no exact picture of softmax activation function?

I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
711 views

What are the best activation functions for Binary text classification in neural networks?

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification. I have heard that Relu is best for Binary ...
21 views

What are contexts where a perceptron model could be defined?

per this post, a perceptron could use the logit function as the activation function. per wiki In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step ...
166 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
33 views

Deep Neural Network and Activation Function

I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in, Increasing model complexity and introduce non-linearity Avoiding Gradient Vanishing ...
435 views

How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for ...
6k views

Using LeakyRelu as activation function in CNN and best alpha for it

Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write: <...
71 views

Confusion regarding the Working mechanism of Activation function

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...
512 views

Why do we use a softmax activation function in Convolutional Autoencoders?

I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras. At the output layer, the author has used the ...
1k views

Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the ...
23 views

How to adjust the Regression with ANN for last part of function

The Blue dots represent the required function and the Black ones represent the predicted function I am using keras with the following code:- ...
2k views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
73 views

Quasi-linearity in deep learning regression problems (sports betting)

I’m attempting to build a sports betting model that aims to predict final scores for games. I’ve had some promising early results for US college football just by using linear regression to form team ...
3k views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
19k views

225 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
46 views

Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as ...
64 views

Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus. My main purpose is to ...
2k views

What does it mean for an activation function to be “saturated/non-saturated”?

For context, in this paper Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. ...
In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...