Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

1
vote
1answer
21 views

Confusion regarding the Working mechanism of Activation function

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted ...
1
vote
1answer
18 views

Why do we use a softmax activation function in Convolutional Autoencoders?

I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras. At the output layer, the author has used the ...
0
votes
0answers
13 views

What's the Advantage of Mutating the Activation Function?

I'm using NEAT (NeuroEvolution of Augmenting Technologies) as a genetic algorithm to evolve my neural network. One of the options in the configuration file for the python implementation of NEAT is to ...
3
votes
1answer
37 views

Exponential Linear Units (ELU) vs $log(1+e^x)$ as the activation functions of deep learning

It seems ELU (Exponential Linear Units) is used as an activation function for deep learning. But its' graph is very similar to the graph of $log(1+e^x)$. So why has $log(1+e^x)$ not been used as the ...
0
votes
1answer
15 views

How to adjust the Regression with ANN for last part of function

The Blue dots represent the required function and the Black ones represent the predicted function I am using keras with the following code:- ...
0
votes
0answers
26 views

activation functions in CNN [duplicate]

I'm new in CNN and I haven't really understood the need of using activation functions such as relu in CNN can anyone explain
7
votes
4answers
258 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
35 views

Quasi-linearity in deep learning regression problems (sports betting)

I’m attempting to build a sports betting model that aims to predict final scores for games. I’ve had some promising early results for US college football just by using linear regression to form team ...
3
votes
0answers
42 views

Gradient Descent in ReLU Neural Network

I’m new to machine learning and recently facing a problem on back propagation of training a neural network using ReLU activation function shown in the figure. My problem is to update the weights ...
11
votes
2answers
938 views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^...
0
votes
0answers
43 views
2
votes
2answers
71 views

What activation function should I use for a specific regression problem?

Which is better for regression problems create a neural net with tanh/sigmoid and exp(like) activations or ReLU and linear? Standard is to use ReLU but it's brute force solution that requires certain ...
2
votes
1answer
26 views

Can we use ReLU activation function as the output layer's non-linearity?

I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value. Can I use ...
2
votes
1answer
27 views

Are there any activation functions which on inputting integer data will produce the output as integers?

Idea is to create the model for ethereum mining which deals with only integer data.
2
votes
2answers
21 views

activation functions in multiple layers in CNNs

An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation ...
3
votes
1answer
32 views

Product of dot products in neural network

In a neural network, it is common to compute a dot product of the form $$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$ and use it as argument to some activation function. This is ...
3
votes
1answer
42 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
1
vote
0answers
23 views

Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as ...
0
votes
0answers
11 views

Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus. My main purpose is to ...
0
votes
1answer
28 views

What does it mean for an activation function to be “saturated/non-saturated”?

For context, in this paper Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. ...
1
vote
0answers
175 views

Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
2
votes
1answer
225 views

Implementing a custom hard sigmoid function

I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values: 0 if x < 0 1 if x > 1 x if 0 <= x <= 1 How do I implement it with a ...
1
vote
1answer
121 views

The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type ...
3
votes
3answers
2k views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
2
votes
1answer
47 views

“Each agent was evaluated every 250,000 training frames for 135,000 validation frames” What does this sentences stands for? in DQN nature paper?

In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions? 0- What is the meaning of this ...
5
votes
2answers
6k views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
1
vote
1answer
130 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
2
votes
1answer
23 views

Restricting the output of a model didn't improve the loss value of the model evaluation

There is a deep model for prediction. The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80) The model Loss value is 70 and I would like to reduce it. I printed the ...
1
vote
1answer
521 views

Neural network example not working with sigmoid activation function

I'm running the Neural Network example written in in BogoToBogo The program worked fine: ...
2
votes
1answer
1k views

Why is the softmax function often used as activation function of output layer in classification neural networks?

What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
1
vote
0answers
33 views

Rules of thumb relating input range of values to choice of activation function

I would like to check with the experts on some observations I made about input value range and choice of activation function in deep learning neural networks. Here are some rules of thumbs I have; ...
3
votes
1answer
118 views

How can ReLU ever fit the curve of x²?

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations: Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ ...
1
vote
1answer
32 views

How to display the value of activation?

I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "...
1
vote
2answers
31 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
0answers
265 views

Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...
3
votes
3answers
1k views

Activation function vs Squashing function

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer. Today in a video explaining deep neural networks I came across the term Squashing ...
0
votes
1answer
147 views

Properly using activation functions of neural network

I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear. Input Layer: From my ...
2
votes
1answer
432 views

Alternatives to linear activation function in regression tasks to limit the output

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
2
votes
0answers
79 views

Cross error loss function cause division by zero error

How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by: -(ylog(actual_output)+(1-y)*(1-...
1
vote
1answer
848 views

Is it possible to customize the activation function in scikit-learn's MLPRegressor?

Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway. Is it possible to change the activation function of the output layer in an MLPRegressor neural ...
1
vote
2answers
56 views

Weights in neural network

So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data ...
1
vote
2answers
71 views

Homemade deep learning library: numerical issue with relu activation

For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade. It seems to work ...
1
vote
1answer
191 views

What exactly is the “hyperbolic” tanh function used in the context of activation functions?

I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows $$ \tanh(x) = \frac{\sinh(x)}{...
1
vote
0answers
35 views

How to Implement Biological Neuron Activations in Artificial Neural Networks

In artificial neural networks, activation functions are used for neurons, i.e. the sigmoid activation: Which can be implemented in code as (in Python): ...
3
votes
2answers
116 views

Is classifier able to say there's no-such-case?

I am a starter in ML and I need some help... The problem Assume that I have a classifier which can classify left hand / right hand well. I am curious whether it can decide whether there's a hand in ...
15
votes
2answers
8k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
1
vote
1answer
2k views

Negative Rewards and Activation Functions

I have a question regarding appropriate activation functions with environments that have both positive and negative rewards. In reinforcement learning, our output, I believe, should be the expected ...
3
votes
1answer
445 views

Is there a way to set a different activation function for each hidden unit in one layer in keras?

I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
4
votes
1answer
5k views

Input normalization for ReLu?

Let's assume a vanilla MLP for classification with a given activation function for hidden layers. I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
4
votes
2answers
362 views

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks? Do they perform better than non-monotonic ones? Is this mathematically proven? Are ...