The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
1
vote
1answer
20 views

Combining multiple neural networks with different activation functions

I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the ...
0
votes
1answer
16 views

ReLU for combating the problem of vanishing gradient in RNN?

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used. When we talk about solving the vanishing gradient problem in RNN, we use a more ...
0
votes
0answers
15 views

Modulo as activation function in neural network

Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our ...
2
votes
2answers
142 views

How to create custom Activation functions in Keras / TensorFlow?

I'm using keras and I wanted to add my own activation function myf to tensorflow backend. how to define the new function and make it operational. so instead of the line of code: ...
0
votes
0answers
10 views

Effect of ReLu derivative in convolution layer backpropagation

I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images. Following this ...
0
votes
0answers
16 views

How exactly should i use softmax activation in last layer of a Neural Network

I am developing a digit classifer with MNIST Dataset. I have read that for classification problems softmax activation function is used, as it maps last layer neuron's outputs into probabilities. ...
0
votes
3answers
73 views

Why there is no exact picture of softmax activation function?

I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
0
votes
1answer
62 views

What are the best activation functions for Binary text classification in neural networks?

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification. I have heard that Relu is best for Binary ...
0
votes
0answers
11 views

Piecewise-linear activation function definition doesn't correspond to graph found in textbook

This question is a duplicate of this question on StackOverflow, but it hasn't been answered in 2 years, which is why I'm re-posting it. If this isn't an appropriate way to go about this, please let me ...
0
votes
0answers
11 views

What are contexts where a perceptron model could be defined?

per this post, a perceptron could use the logit function as the activation function. per wiki In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step ...
0
votes
1answer
25 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
0
votes
0answers
21 views

Thresholding in intermediate layer using Gumbel Softmax

In a neural network, for an intermediate layer, I need to threshold the output. The output of each neuron in the layer is a real value, but I need to binarize it (to 0 or 1). But with hard ...
0
votes
1answer
20 views

Deep Neural Network and Activation Function

I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in, Increasing model complexity and introduce non-linearity Avoiding Gradient Vanishing ...
0
votes
3answers
33 views

How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for ...
1
vote
1answer
691 views

Using LeakyRelu as activation function in CNN and best alpha for it

Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write: <...
1
vote
1answer
33 views

Confusion regarding the Working mechanism of Activation function

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...
1
vote
2answers
85 views

Why do we use a softmax activation function in Convolutional Autoencoders?

I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras. At the output layer, the author has used the ...
0
votes
0answers
18 views

What's the Advantage of Mutating the Activation Function?

I'm using NEAT (NeuroEvolution of Augmenting Technologies) as a genetic algorithm to evolve my neural network. One of the options in the configuration file for the python implementation of NEAT is to ...
4
votes
1answer
95 views

Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the ...
1
vote
1answer
19 views

How to adjust the Regression with ANN for last part of function

The Blue dots represent the required function and the Black ones represent the predicted function I am using keras with the following code:- ...
0
votes
0answers
38 views

activation functions in CNN [duplicate]

I'm new in CNN and I haven't really understood the need of using activation functions such as relu in CNN can anyone explain
7
votes
4answers
367 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
45 views

Quasi-linearity in deep learning regression problems (sports betting)

I’m attempting to build a sports betting model that aims to predict final scores for games. I’ve had some promising early results for US college football just by using linear regression to form team ...
4
votes
2answers
123 views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
16
votes
2answers
4k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^...
0
votes
0answers
280 views

What are best activation and regularization method for LSTM?

In Keras there are: ...
2
votes
2answers
166 views

What activation function should I use for a specific regression problem?

Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain ...
2
votes
1answer
62 views

Can we use ReLU activation function as the output layer's non-linearity?

I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value. Can I use ...
2
votes
1answer
29 views

Are there any activation functions which on inputting integer data will produce the output as integers?

Idea is to create the model for ethereum mining which deals with only integer data.
2
votes
2answers
26 views

activation functions in multiple layers in CNNs

An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation ...
3
votes
1answer
42 views

Product of dot products in neural network

In a neural network, it is common to compute a dot product of the form $$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$ and use it as argument to some activation function. This is ...
3
votes
1answer
65 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
1
vote
0answers
32 views

Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as ...
1
vote
0answers
25 views

Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus. My main purpose is to ...
0
votes
1answer
60 views

What does it mean for an activation function to be “saturated/non-saturated”?

For context, in this paper Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. ...
1
vote
1answer
253 views

Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
2
votes
1answer
443 views

Implementing a custom hard sigmoid function

I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values: 0 if x < 0 1 if x > 1 x if 0 <= x <= 1 How do I implement it with a ...
1
vote
1answer
197 views

The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type ...
3
votes
3answers
4k views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
2
votes
1answer
50 views

“Each agent was evaluated every 250,000 training frames for 135,000 validation frames” What does this sentences stands for? in DQN nature paper?

In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions? 0- What is the meaning of this ...
8
votes
2answers
12k views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
1
vote
1answer
224 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
2
votes
1answer
26 views

Restricting the output of a model didn't improve the loss value of the model evaluation

There is a deep model for prediction. The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80) The model Loss value is 70 and I would like to reduce it. I printed the ...
1
vote
1answer
781 views

Neural network example not working with sigmoid activation function

I'm running the Neural Network example written in in BogoToBogo The program worked fine: ...
2
votes
1answer
2k views

Why is the softmax function often used as activation function of output layer in classification neural networks?

What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
1
vote
0answers
42 views

Rules of thumb relating input range of values to choice of activation function

I would like to check with the experts on some observations I made about input value range and choice of activation function in deep learning neural networks. Here are some rules of thumbs I have; ...
3
votes
1answer
169 views

How can ReLU ever fit the curve of x²?

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations: Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ ...
1
vote
1answer
36 views

How to display the value of activation?

I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "...
2
votes
4answers
50 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
0answers
345 views

Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...