All Questions
Tagged with neural-network activation-function
91 questions
0
votes
1
answer
23
views
Why does tanh activation work better with Pytorch than with Keras?
I'm doing a neural network to recognize written Cyrillic letters, and I found out that, when I use tanh activation function, it works WAY better with PyTorch than with Keras.
Keras code:
...
- 1
0
votes
0
answers
46
views
Why do we use the RELU activation function?
I reading about activation functions in feedforward neural networks.
ad a really old paper https://web.njit.edu/~usman/courses/cs677_spring21/hornik-nn-1991.pdf.
They prove that by using arbitrary ...
- 76
1
vote
0
answers
24
views
What is a proper activation function with simulated-annealing trainer for neural network?
I'm developing a gpu-accelerated simulated annealing based neural network trainer library. Currently its stuck on how to converge on "array sorting by neural network 3:10:20:10:3 topology".
...
1
vote
1
answer
961
views
Whats the advantage of He Intialization over Xavier Intialization?
For Weights initialization, I read that He doesn't consider linear activation of neurons as Xavier Initialization; in this context, what does linear initialization mean?
- 13
0
votes
1
answer
50
views
Is it possible to tell if one activation function is better than the other one based on their graphs?
I am attempting to formulate my own activation function. However, I'm new to neural networks, am not yet ready to test it, but would want to know if I already landed on a better activation function ...
- 3
2
votes
1
answer
63
views
Question about non linearity of activation function
I have a basic question about activation functions. It is told that they are added to the network to introduce non linearity.
However, the neural network itself is non linear. Isn' it? If we see any ...
- 914
0
votes
1
answer
349
views
Training deep neural networks with ReLU output layer for verification
Most algorithms for verification of deep neural network require ReLU activation functions in each layer (e.g. Reluplex).
I have a binary classification task with classes 0 and 1. The main problem I ...
- 1
5
votes
0
answers
189
views
Intuitively, why do Non-monotonic Activations Work?
The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
- 73
0
votes
1
answer
18
views
Activation Functions in Haykins Neural Networks a comprehensive foundation
In Haykins Neural Network a comprehensive foundation, the piecwise-linear funtion is one of the described activation functions.
It is described with:
The corresponding shown plot is
I don't really ...
0
votes
1
answer
412
views
Why does using tanh worsen accuracy so much?
I was testing how different hyperparameters would change the output of my multilayer perceptron for a regression problem
...
3
votes
4
answers
1k
views
Neural Network not Deep
I have found this image link I would like to know what NNs are not deep neural? The first three?
Also what kind of functional activations do they use?
- 491
2
votes
1
answer
623
views
How to prove Softmax Numerical Stability?
I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of ...
- 21
0
votes
0
answers
31
views
What happens if you don't include any activation function on hidden classification layers?
What happens if we don't apply an activation function to the classification hidden layers and apply it only for the final output layer (Sigmoid, Softmax)?
I'm asking this because I have trained a CNN ...
- 27
0
votes
2
answers
214
views
Activation and Loss Function not chosen correctly when use Neural Network
I have three classes for my text dataset before.
These are my classes:
0 = Cat
1 = Not Both
2 = Dog
Then I use this code:
...
- 13
2
votes
0
answers
103
views
Derive backpropagation for PreLU
I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows:
$$
h_a(x) = \text{max}(ax, x)
$$
I want to derive $ \frac{\partial L}{\partial ...
- 21
19
votes
2
answers
12k
views
Why deep learning models still use RELU instead of SELU, as their activation function?
I am a trying to understand the SELU activation function and I was wondering why deep learning practitioners keep using RELU, with all its issues, instead of SELU, which enables a neural network to ...
10
votes
2
answers
25k
views
ReLU vs Leaky ReLU vs ELU with pros and cons
I am unable to understand when to use ReLU, Leaky ReLU and ELU.
How do they compare to other activation functions(like the sigmoid and the tanh) and their pros and cons.
- 113
1
vote
1
answer
218
views
Activation Function
I am very new to machine learning and made an experiment myself. I have a few questions:
Can I use $Y = sin(x)$ or $Y = 2x$ as an activation function for a neural network?
Is it necessary to increase ...
- 13
1
vote
2
answers
240
views
Is there comprehensive list of activation functions and their applications for a Neural Network?
I am aware of common activation functions like Sigmoid, Tanh, ReLU, Leaky ReLU. Even heard about a function called Swish(SiLU). Now is there any detailed information on other activations functions and ...
- 11
0
votes
1
answer
217
views
Applying activation on part of the layer in Keras
Context
I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network:
...
- 101
0
votes
1
answer
177
views
Input and output layer activation functions of neurons in Orange
What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?
- 1
0
votes
1
answer
27
views
Can there be in-active neuron in output layer
I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a ...
- 163
1
vote
1
answer
105
views
Are non-relu activations better for small/ dense datasets?
Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity.
Why is ...
- 539
0
votes
1
answer
189
views
Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?
I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, ...
- 103
0
votes
1
answer
225
views
How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?
I was looking into the multi-label classification on the output layer of a Neural Network.
I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons.
So for example an ...
1
vote
1
answer
1k
views
Problem with convergence of ReLu in MLP
I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens.
...
2
votes
2
answers
429
views
Dying leaky ReLU
I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
3
votes
1
answer
876
views
Why the sigmoid activation function results in sub-optimal gradient descent?
I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
- 33
2
votes
1
answer
2k
views
As RELU is not differentiable when it touches the x-axis, doesn't it effect training?
When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent.
I am a beginner in deep ...
- 709
1
vote
0
answers
71
views
Sharing parameters of an activation across layers of a neural network
Keras now provides advanced parametric activation layers like Leaky-ReLU PReLU.
Each time I add this layer to a sequential model, an additional trainable parameter is added to graph.
How can I make ...
- 163
1
vote
1
answer
1k
views
How does Pytorch deal with non-differentiable activation functions during backprop?
I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
- 123
1
vote
0
answers
29
views
Wich activation function for DQL
After many research, I still can't find a neat answer about this question:
When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
4
votes
3
answers
2k
views
What does the descision boundary of a relu look like?
A single non activated neuron is just a linear combination of its inputs.
Thresholding this neuron's output as-is against 0 would create a hyperplane binary separator, whose parameters can be learned....
- 196
11
votes
3
answers
5k
views
Why Leaky ReLU is not so common in real practice?
As Leaky ReLU does not lead any value to 0, so training always continues. And I can't think of any disadvantages it has.
Yet Leaky ReLU is less popular than ReLU in real practice. Can someone tell why ...
- 221
1
vote
1
answer
247
views
Example of convex activation function
For a particular task, I need a convex activation function with the following properties:
f''(x) > 0
0 <= f(x) <= 1
f(x) is monotonic
f(x) is not "exploding" i.e. avoiding functions such as ...
- 111
2
votes
1
answer
3k
views
Different activation function in same layer of a Neural network
My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers.
Suppose I have 3 ...
- 143
1
vote
0
answers
273
views
Generalized softmax derivative for implementation with any loss function
I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
- 11
1
vote
0
answers
104
views
Which activation function of the output layer and which loss function are advised to be used for bounded regression?
I want my (deep) neural network to produce an output from a certain range, in my case between 0 and 255. I have scaled the labels from [0..255] to [0..1].
For the neural network, I have tried a ...
- 111
1
vote
0
answers
929
views
Writing a piecewise linear function as a sum of ReLU functions
Suppose I have a piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, where $\{\phi_i\}_{i=1}^n$ is a finite dimension space of dimension $n-1$, in particular I am interested in the functions ...
- 111
5
votes
2
answers
622
views
What came first? Backpropagation or Sigmoid?
Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function.
When did the sigmoid function become so popular in NNs?
- 419
3
votes
1
answer
1k
views
Leaky ReLU inside of a Simple Python Neural Net
To build a simple 1-layer neural network, many tutorials use a sigmoid function as the activation function. According to scholarly articles and other online sources, a leaky ReLU is a better ...
- 133
1
vote
0
answers
27
views
How does one use activation function with greater than [-1;1] range for binary classification?
In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes.
They explain (§ 4....
- 2,444
8
votes
2
answers
612
views
How does one derive the modified tanh activation proposed by LeCun?
In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form:
$$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$
...
- 2,444
7
votes
1
answer
3k
views
Is it wrong to use Glorot Initialization with ReLu Activation?
I'm reading that keras' default initialization is glorot_uniform.
However, all of the tutorials I see are using relu ...
- 539
3
votes
1
answer
38
views
How to quantitatively evaluate raw neural network activations?
Below are the activations for 2 different predictions. These predictions are for different labels/classes. They are being run through a dense Keras NN (96.6% accurate) with 2 hidden layers and adam ...
- 539
1
vote
1
answer
1k
views
Counting Number of Parameters in Neural Networks [closed]
Note: This is an academics based problem.
So in a recent in-class quiz, we were asked that if we have an input layer consisting of 20 nodes along with 2 hidden layers (one of size 10 and the other of ...
- 191
1
vote
0
answers
112
views
Should I scale/normalize the data before training a feedforward neural network using only lagged values?
So I am trying to forecast the price of a certain commodity that is quite volatile. Now I wanted to train a regression feedforward neural network to predict the price of next week.
Just to be clear, ...
- 11
2
votes
1
answer
2k
views
Relationship between Sigmoid and Gaussing Distribution
I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?":
The assumption of a dependent variable to follow ...
- 21
-1
votes
2
answers
87
views
Activation Functions in Neural network
I have a set of questions related to the usage of various activation functions used in neural networks. I would highly appreciate if someone could give explanatory answers.
Why is ReLU is used only ...
- 21
3
votes
2
answers
775
views
Why activation functions used in neural networks generally have limited range?
Why do we generally use activation functions with only limited range in neural networks? for e.g.
$sigmoid$ activation function has range $[0, 1]$
$tanh$ activation function has range $[-1, 1]$
Q1) ...
- 175