Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Effect of ReLu derivative in convolution layer backpropagation

I'm trying to implement a CNN, as part of an academic project to learn how it works. The project is a SRCNN: a convolutional neural network that increase the resolution of images. Following this ...
0
votes
0answers
12 views

How exactly should i use softmax activation in last layer of a Neural Network

I am developing a digit classifer with MNIST Dataset. I have read that for classification problems softmax activation function is used, as it maps last layer neuron's outputs into probabilities. ...
0
votes
3answers
35 views

Why there is no exact picture of softmax activation function?

I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
0
votes
1answer
17 views

What are the best activation functions for Binary text classification in neural networks?

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification. I have heard that Relu is best for Binary ...
0
votes
0answers
9 views

Piecewise-linear activation function definition doesn't correspond to graph found in textbook

This question is a duplicate of this question on StackOverflow, but it hasn't been answered in 2 years, which is why I'm re-posting it. If this isn't an appropriate way to go about this, please let me ...
0
votes
0answers
11 views

What are contexts where a perceptron model could be defined?

per this post, a perceptron could use the logit function as the activation function. per wiki In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step ...
0
votes
1answer
17 views

Derivative of activation function used in gradient descent algorithms

Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a ...
0
votes
0answers
13 views

Thresholding in intermediate layer using Gumbel Softmax

In a neural network, for an intermediate layer, I need to threshold the output. The output of each neuron in the layer is a real value, but I need to binarize it (to 0 or 1). But with hard ...
0
votes
1answer
20 views

Deep Neural Network and Activation Function

I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in, Increasing model complexity and introduce non-linearity Avoiding Gradient Vanishing ...
0
votes
3answers
29 views

How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for ...
1
vote
1answer
278 views

Using LeakyRelu as activation function in CNN and best alpha for it

Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write: <...
1
vote
1answer
28 views

Confusion regarding the Working mechanism of Activation function

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...
1
vote
2answers
47 views

Why do we use a softmax activation function in Convolutional Autoencoders?

I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras. At the output layer, the author has used the ...
0
votes
0answers
16 views

What's the Advantage of Mutating the Activation Function?

I'm using NEAT (NeuroEvolution of Augmenting Technologies) as a genetic algorithm to evolve my neural network. One of the options in the configuration file for the python implementation of NEAT is to ...
4
votes
1answer
59 views

Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the ...
1
vote
1answer
19 views

How to adjust the Regression with ANN for last part of function

The Blue dots represent the required function and the Black ones represent the predicted function I am using keras with the following code:- ...
0
votes
0answers
32 views

activation functions in CNN [duplicate]

I'm new in CNN and I haven't really understood the need of using activation functions such as relu in CNN can anyone explain
7
votes
4answers
291 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
38 views

Quasi-linearity in deep learning regression problems (sports betting)

I’m attempting to build a sports betting model that aims to predict final scores for games. I’ve had some promising early results for US college football just by using linear regression to form team ...
4
votes
0answers
68 views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
12
votes
2answers
2k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^...
0
votes
0answers
155 views
2
votes
2answers
117 views

What activation function should I use for a specific regression problem?

Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain ...
2
votes
1answer
33 views

Can we use ReLU activation function as the output layer's non-linearity?

I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value. Can I use ...
2
votes
1answer
28 views

Are there any activation functions which on inputting integer data will produce the output as integers?

Idea is to create the model for ethereum mining which deals with only integer data.
2
votes
2answers
23 views

activation functions in multiple layers in CNNs

An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation ...
3
votes
1answer
36 views

Product of dot products in neural network

In a neural network, it is common to compute a dot product of the form $$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$ and use it as argument to some activation function. This is ...
3
votes
1answer
55 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
1
vote
0answers
29 views

Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as ...
1
vote
0answers
19 views

Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus. My main purpose is to ...
0
votes
1answer
37 views

What does it mean for an activation function to be “saturated/non-saturated”?

For context, in this paper Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. ...
1
vote
1answer
229 views

Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
2
votes
1answer
345 views

Implementing a custom hard sigmoid function

I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values: 0 if x < 0 1 if x > 1 x if 0 <= x <= 1 How do I implement it with a ...
1
vote
1answer
177 views

The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type ...
3
votes
3answers
3k views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
2
votes
1answer
49 views

“Each agent was evaluated every 250,000 training frames for 135,000 validation frames” What does this sentences stands for? in DQN nature paper?

In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions? 0- What is the meaning of this ...
7
votes
2answers
9k views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
1
vote
1answer
184 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
2
votes
1answer
24 views

Restricting the output of a model didn't improve the loss value of the model evaluation

There is a deep model for prediction. The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80) The model Loss value is 70 and I would like to reduce it. I printed the ...
1
vote
1answer
644 views

Neural network example not working with sigmoid activation function

I'm running the Neural Network example written in in BogoToBogo The program worked fine: ...
2
votes
1answer
2k views

Why is the softmax function often used as activation function of output layer in classification neural networks?

What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
1
vote
0answers
38 views

Rules of thumb relating input range of values to choice of activation function

I would like to check with the experts on some observations I made about input value range and choice of activation function in deep learning neural networks. Here are some rules of thumbs I have; ...
3
votes
1answer
141 views

How can ReLU ever fit the curve of x²?

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations: Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ ...
1
vote
1answer
34 views

How to display the value of activation?

I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "...
1
vote
3answers
42 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
0answers
306 views

Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...
3
votes
3answers
2k views

Activation function vs Squashing function

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer. Today in a video explaining deep neural networks I came across the term Squashing ...
0
votes
1answer
163 views

Properly using activation functions of neural network

I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear. Input Layer: From my ...
2
votes
1answer
502 views

Alternatives to linear activation function in regression tasks to limit the output

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
2
votes
0answers
80 views

Cross error loss function cause division by zero error

How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by: -(ylog(actual_output)+(1-y)*(1-...