Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
25
votes
2answers
14k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
21
votes
2answers
9k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + ...
17
votes
1answer
21k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
15
votes
1answer
14k views

Difference of Activation Functions in Neural Networks in general

I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear. It's reasonable that one ...
11
votes
3answers
20k views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
7
votes
4answers
691 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
6
votes
2answers
4k views

What is the purpose of multiple neurons in a hidden layer?

On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer. Let me make the question more clear. Take this ...
5
votes
1answer
212 views

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
5
votes
3answers
3k views

Activation function vs Squashing function

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer. Today in a video explaining deep neural networks I came across the term Squashing ...
5
votes
3answers
3k views

How to create custom Activation functions in Keras / TensorFlow?

I'm using keras and I wanted to add my own activation function myf to tensorflow backend. how to define the new function and make it operational. so instead of the line of code: ...
5
votes
2answers
516 views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
4
votes
6answers
877 views

Why is activation needed at all in neural network?

I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know ...
4
votes
3answers
6k views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
4
votes
1answer
8k views

Input normalization for ReLu?

Let's assume a vanilla MLP for classification with a given activation function for hidden layers. I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
4
votes
2answers
533 views

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks? Do they perform better than non-monotonic ones? Is this mathematically proven? Are ...
4
votes
1answer
1k views

Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the ...
3
votes
2answers
124 views

Is classifier able to say there's no-such-case?

I am a starter in ML and I need some help... The problem Assume that I have a classifier which can classify left hand / right hand well. I am curious whether it can decide whether there's a hand in ...
3
votes
3answers
3k views

Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects: simple more biologically plausible no gradient to vanish better performance sparsity And I see only one ...
3
votes
2answers
39 views

What came first? Backpropagation or Sigmoid?

Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function. When did the sigmoid function become so popular in NNs?
3
votes
1answer
35 views

Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is used as a an activation function which serve two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem we use Gradient Clipping ...
3
votes
1answer
51 views

Leaky ReLU inside of a Simple Python Neural Net

To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better ...
3
votes
1answer
412 views

Can we use ReLU activation function as the output layer's non-linearity?

I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value. Can I use ...
3
votes
2answers
62 views

Why activation functions used in neural networks generally have limited range?

Why do we generally use activation functions with only limited range in neural networks? for e.g. $sigmoid$ activation function has range $[0, 1]$ $tanh$ activation function has range $[-1, 1]$ Q1) ...
3
votes
1answer
25 views

How to quantitatively evaluate raw neural network activations?

Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam ...
3
votes
2answers
205 views

TensorFlow Sigmoid activation function as output layer - value interpretation

My TensorFlow model has the following structure. It aims to solve a binary classification problem where the labels are either 0 or 1. The output layer uses a sigmoid activation function with 1 output. ...
3
votes
1answer
61 views

Product of dot products in neural network

In a neural network, it is common to compute a dot product of the form $$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$ and use it as argument to some activation function. This is ...
3
votes
1answer
97 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
3
votes
1answer
251 views

How can ReLU ever fit the curve of x²?

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations: Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ ...
3
votes
1answer
591 views

Is there a way to set a different activation function for each hidden unit in one layer in keras?

I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
3
votes
0answers
32 views

How does one derive the modified tanh activation proposed by LeCun?

In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form: $$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$ ...
3
votes
1answer
57 views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
2
votes
2answers
535 views

What activation function should I use for a specific regression problem?

Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain ...
2
votes
1answer
2k views

Using LeakyRelu as activation function in CNN and best alpha for it

Since if we do not declare the activation function, the default will be set as linear for Conv2D layer. Is it true to write: <...
2
votes
1answer
802 views

Alternatives to linear activation function in regression tasks to limit the output

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
2
votes
1answer
141 views

Why is an activation function notated as “g”?

In many cases an activation function is notated as g (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as ...
2
votes
2answers
50 views

activation functions in multiple layers in CNNs

An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation ...
2
votes
1answer
3k views

Why is the softmax function often used as activation function of output layer in classification neural networks?

What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
2
votes
1answer
63 views

Are there any activation functions which on inputting integer data will produce the output as integers?

Idea is to create the model for ethereum mining which deals with only integer data.
2
votes
1answer
780 views

Implementing a custom hard sigmoid function

I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values: 0 if x < 0 1 if x > 1 x if 0 <= x <= 1 How do I implement it with a ...
2
votes
1answer
51 views

“Each agent was evaluated every 250,000 training frames for 135,000 validation frames” What does this sentences stands for? in DQN nature paper?

In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions? 0- What is the meaning of this ...
2
votes
1answer
27 views

Restricting the output of a model didn't improve the loss value of the model evaluation

There is a deep model for prediction. The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80) The model Loss value is 70 and I would like to reduce it. I printed the ...
2
votes
1answer
45 views

Counting Number of Parameters in Neural Networks

Note: This is an academics based problem. So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of ...
2
votes
0answers
81 views

Cross error loss function cause division by zero error

How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by: -(ylog(actual_output)+(1-y)*(1-...
2
votes
4answers
60 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
2answers
85 views

Weights in neural network

So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data ...
1
vote
2answers
92 views

Homemade deep learning library: numerical issue with relu activation

For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade. It seems to work ...
1
vote
1answer
63 views

Binary classifier using Keras with backend Tensorflow with a Binary output

I am trying to build a binary classifier with tensorflow.keras Currently unable to identify a solution to having the model generating only 0s and 1s. The code for compiling my tensorflow model. <...
1
vote
1answer
51 views

Confusion regarding the Working mechanism of Activation function

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...
1
vote
1answer
2k views

Is it possible to customize the activation function in scikit-learn's MLPRegressor?

Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway. Is it possible to change the activation function of the output layer in an MLPRegressor neural ...
1
vote
1answer
251 views

What exactly is the “hyperbolic” tanh function used in the context of activation functions?

I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows $$ \tanh(x) = \frac{\sinh(x)}{...