Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
0 votes
0 answers
32 views

Why do we use the RELU activation function?

I reading about activation functions in feedforward neural networks. ad a really old paper https://web.njit.edu/~usman/courses/cs677_spring21/hornik-nn-1991.pdf. They prove that by using arbitrary ...
timmy1691's user avatar
1 vote
1 answer
48 views

How does ReLU function make it possible to let the CNN learn more complex features in input data?

In many descriptions of a CNN i often read that at the end of the Convolutional layer, a ReLU function is needed, for two reasons: first it solves many problems about the vanishing gradient problem, ...
J. Drawman's user avatar
0 votes
0 answers
25 views

Why does scaled dot-product attention use softmax?

I am trying to understand the reasoning behind the Transformer architecture. In "Attention is all you need", the weights for the scaled dot-product attention is defined as the scaled dot-...
Reinis Mazeiks's user avatar
0 votes
3 answers
86 views

Why Relu is correct for CNN?

Relu only passes positive values, so when we calculate the gradients for this layer, we will only get positive gradients. The gradient for the filter weights of this layer is the convolution of the ...
Tima's user avatar
  • 1
1 vote
0 answers
12 views

What is a proper activation function with simulated-annealing trainer for neural network?

I'm developing a gpu-accelerated simulated annealing based neural network trainer library. Currently its stuck on how to converge on "array sorting by neural network 3:10:20:10:3 topology". ...
huseyin tugrul buyukisik's user avatar
1 vote
1 answer
33 views

Why would multiple activation layers be used in a row?

I'm learning about ML and I was looking at ...
Thomas's user avatar
  • 155
0 votes
0 answers
64 views

Alternative to ELU and Leaky ReLU?

I was talking with a friend about different activation functions (we are still new to ML). One thing that I didn't like about ELU was the vanishing gradient, and about Leaky ReLU that it's not ...
Nasa's user avatar
  • 1
0 votes
0 answers
25 views

Relu derivative value

I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
Gunners 's user avatar
0 votes
1 answer
30 views

Query about Sigmoid activation function calculation

While applying sigmoid activation function (in finding y label), I have calculated it as below: y = 0.35 + (0.8 * 0.1) + (0.3 * 0.6) + (-0.2 * 0.4) = 0.53 sigmoid_y = 0.625 how do we take threshold ...
tovijayak's user avatar
0 votes
1 answer
50 views

How to implement a custom loss with a non-mathematical operation (simulation) that backpropagates with PyTorch?

I am writing a Neural Network, which output is not used directly for the loss-function, but rather as the input for a simulation model. After the simulation ran, I am using the simulated_value and the ...
zenia's user avatar
  • 1
0 votes
1 answer
164 views

How ReLU is bringing non linearity and why it is not an alternative to dropout?

The differentiation of ReLU function is 1 when input is greater than 0, and 0, when input is less than or equal to 0. In the backpropagation process it doesn’t change the value of d(error)/d(weight) ...
mir abir hossain's user avatar
0 votes
2 answers
332 views

Why use tanh (or any other activation function)?

In machine learning, it is common to use activation functions like tanh, sigmoid, or ReLU to introduce non-linearity into a neural network. These non-linearities help the network learn complex ...
good_evening's user avatar
0 votes
3 answers
204 views

Regression model doesn't handle negative values

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation. To do this, I'm using the ReLU activation function for each layer. The ...
Iya Lee's user avatar
  • 152
0 votes
1 answer
524 views

Whats the advantage of He Intialization over Xavier Intialization?

For Weights initialization, I read that He doesn't consider linear activation of neurons as Xavier Initialization; in this context, what does linear initialization mean?
Carpediem's user avatar
0 votes
1 answer
18 views

Why is the optimal output out of domain in A2C?

If each state has an optimal action, then the optimal actions distribution vector is a one-hot vector kind of like [0,0,1,0,0,0]. But with algorithms like A2C, we ...
RyuGood0's user avatar
0 votes
0 answers
64 views

Sigmoid Activation Function (Output layer) Alternative

I have a Convolutional-VAE architecture where the target images are in the range [0, 1], their pixel values. To synthesize/reconstruct images in this scale, I am using a sigmoid activation function in ...
Arun's user avatar
  • 121
0 votes
1 answer
34 views

Is it possible to tell if one activation function is better than the other one based on their graphs?

I am attempting to formulate my own activation function. However, I'm new to neural networks, am not yet ready to test it, but would want to know if I already landed on a better activation function ...
jwho's user avatar
  • 3
0 votes
1 answer
588 views

CNN model why is ReLu used in Conv1D layer and in the first Dense Layer?

I have a problem. I have a CNN model which is used for an NLP problem. This is written in Python. I have questions about this, which I can't find an answer to. Why is ReLu used inside the Conv1D ...
Test's user avatar
  • 89
2 votes
1 answer
53 views

Question about non linearity of activation function

I have a basic question about activation functions. It is told that they are added to the network to introduce non linearity. However, the neural network itself is non linear. Isn' it? If we see any ...
Sandeep Bhutani's user avatar
5 votes
1 answer
246 views

What is the advantage of using Euler's number ($e^x$) instead of another base in the softmax equation?

I understand the softmax equation is $\boldsymbol{P}(y=j \mid x)=\frac{e^{x_{j}}}{\sum_{k=1}^{K} e^{x_{k}}}$ My question is: why use $e^x$ instead of say, $3^x$. I understand $e^x$ is it's own ...
Codedorf's user avatar
0 votes
2 answers
1k views

If sigmoid activation function is prone to vanishing and exploding gradients can we not use it in final layer of binary classfication?

Many paper and books say that sigmoid activation function with random intialization is prone to vanishing/exploding gradients therefore it is better to use LeakyRelu, Elu, or Relu. Does this mean that ...
haneulkim's user avatar
  • 457
0 votes
1 answer
280 views

Training deep neural networks with ReLU output layer for verification

Most algorithms for verification of deep neural network require ReLU activation functions in each layer (e.g. Reluplex). I have a binary classification task with classes 0 and 1. The main problem I ...
alext90's user avatar
0 votes
0 answers
342 views

Performance comparison of different activation functions

I was trying to find a way to compare the test accuracy and test loss of different activation functions (such as tanh, sigmoid, relu), so I came up with this script: ...
Amirhossein Rezaei's user avatar
3 votes
0 answers
122 views

Intuitively, why do Non-monotonic Activations Work?

The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
Jason's user avatar
  • 53
0 votes
1 answer
18 views

Activation Functions in Haykins Neural Networks a comprehensive foundation

In Haykins Neural Network a comprehensive foundation, the piecwise-linear funtion is one of the described activation functions. It is described with: The corresponding shown plot is I don't really ...
DerWolferl's user avatar
0 votes
1 answer
279 views

Why does using tanh worsen accuracy so much?

I was testing how different hyperparameters would change the output of my multilayer perceptron for a regression problem ...
SGfrmthe33's user avatar
3 votes
4 answers
1k views

Neural Network not Deep

I have found this image link I would like to know what NNs are not deep neural? The first three? Also what kind of functional activations do they use?
Inuraghe's user avatar
  • 481
0 votes
0 answers
84 views

Why using the hyperbolic tangent or the sigmoid as activation function on the last layer gaves the same result in accuracy?

The problem I'm making a simple Multilayer Perceptron (MLP), in Keras, that has to do the binary classification from some float type of data. Each single data is a group of three float values (e.g. 32....
Simone Starace's user avatar
1 vote
1 answer
491 views

How to prove Softmax Numerical Stability?

I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of ...
Nicoinlas's user avatar
0 votes
0 answers
27 views

What happens if you don't include any activation function on hidden classification layers?

What happens if we don't apply an activation function to the classification hidden layers and apply it only for the final output layer (Sigmoid, Softmax)? I'm asking this because I have trained a CNN ...
Valderas's user avatar
0 votes
2 answers
432 views

Is there any point in having layers in a neural network for regression problems?

In my textbook I read that an MLP and linear activation functions for the hidden layers can be reduced to a simple input-output system, i.e. no hidden layers. This makes sense to me. Later on I read ...
falxman's user avatar
0 votes
2 answers
189 views

Activation and Loss Function not chosen correctly when use Neural Network

I have three classes for my text dataset before. These are my classes: 0 = Cat 1 = Not Both 2 = Dog Then I use this code: ...
grace's user avatar
  • 13
2 votes
0 answers
62 views

Derive backpropagation for PreLU

I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows: $$ h_a(x) = \text{max}(ax, x) $$ I want to derive $ \frac{\partial L}{\partial ...
Casper's user avatar
  • 21
1 vote
0 answers
32 views

What meaning does exp function have? [closed]

Is there any problem that solution(or algorithm) would be exp function? Let's say f(x)=2^x. It's an answer of a problem when you would like to know how many pieces would be made when you fold a paper ...
sugu's user avatar
  • 21
1 vote
0 answers
23 views

The case of (1,478) dim and parameters of neural network to find out

colleagues, actually I am kind'a new to NN, but hard trying.. I have data: Index: 40073 entries (excluded from training, UID) Columns: 484 entries dtypes: bool(468), float64(2), int64(13), object(1) I ...
Gleb Karpushkin's user avatar
12 votes
2 answers
7k views

Why deep learning models still use RELU instead of SELU, as their activation function?

I am a trying to understand the SELU activation function and I was wondering why deep learning practitioners keep using RELU, with all its issues, instead of SELU, which enables a neural network to ...
Konstantinos Skoularikis's user avatar
8 votes
1 answer
19k views

Difference between ReLU, ELU and Leaky ReLU. Their pros and cons majorly

I am unable to understand when to use ReLU, ELU and Leaky ReLU. How do they compare to other activation functions(like the sigmoid and the tanh) and their pros and cons.
Ayazzia01's user avatar
1 vote
0 answers
141 views

Word embedding autoencoder

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated. Here is my ...
user1513335's user avatar
1 vote
1 answer
140 views

Activation Function

I am very new to machine learning and made an experiment myself. I have a few questions: Can I use $Y = sin(x)$ or $Y = 2x$ as an activation function for a neural network? Is it necessary to increase ...
Future's user avatar
  • 13
1 vote
2 answers
119 views

How can the ReLU function lead to convergence?

The gradient descent algorithm is based on the fact that the gradient decreases as we move towards the optimum point. However, in the activations by the ReLU ...
Sanchit Goyal's user avatar
1 vote
0 answers
199 views

If the input to the autoencoder is normalized, do we need to use sigmoid on the last layer?

According to: https://stackoverflow.com/questions/65307833/why-is-the-decoder-in-an-autoencoder-uses-a-sigmoid-on-the-last-layer The last layer activation function contains sigmoid in order to the ...
user3668129's user avatar
1 vote
1 answer
189 views

Is there comprehensive list of activation functions and their applications for a Neural Network?

I am aware of common activation functions like sigmoid, tanh,ReLu, Leaky ReLu. Even heard about a function called Swish. Now is there any detailed information on other activations functions and some ...
yash kumar's user avatar
1 vote
1 answer
99 views

Can this dataset be separated linearly?

Is this dataset linearly separable? If not, can it be converted into one by applying some function as it seems to follow the same pattern? Also, which classification algorithms could be used to fit ...
heerboi's user avatar
  • 13
3 votes
1 answer
5k views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
user3668129's user avatar
0 votes
1 answer
760 views

Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
Boom's user avatar
  • 297
1 vote
1 answer
52 views

Training set Distribution and Activation function/Loss function correlation

How should the probability distribution of the training set influence the choice of the activation function / loss function? For instance if I have a Multinoulli distribution, which activation ...
Turned Capacitor's user avatar
2 votes
1 answer
204 views

Do Activation Functions map to Higher Dimensions?:

I just started learning tensorflow and I have a question regarding activation functions used in neural networks, I watched a 3b1b video a while ago and it seems it squished the value into an interval ...
FoundABetterName's user avatar
0 votes
1 answer
399 views

Numerically stable hyperbolic tangent

The hyperbolic tangent is commonly used as an activation function: $$ tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} $$ Although, it is unclear how this function is implemented to be numerically stable ...
Kevin's user avatar
  • 163
0 votes
1 answer
208 views

Applying activation on part of the layer in Keras

Context I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network: ...
LeeNeverGup's user avatar
0 votes
1 answer
80 views

Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...
Winston's user avatar
  • 103