Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

What is meant by softly selecting a matrices from a set of matrices?

How is the softmax layer applied to the 1x1 conv layer in order to soflty select the matrix from a set of matrix denoted by A. From my understanding of convolutions, implementing a 1x1 conv layer on a ...
user avatar
0 votes
0 answers
12 views

Intuitively, why do Non-monotonic Activations Work?

The swish/SiLU activation is very popular, and many would argue it has dethroned ReLU. However, it is non-monotonic, which seems to go against popular intuition (at least on this site: example 1, ...
user avatar
  • 23
0 votes
0 answers
9 views

why zero centring of data from activation function is good for deep nural network?

I was reading an article that mentioned reasons why tanh is better than sigmoid and one reason was that tanh gives zero-centered data but I couldn't understand why and how it will affect our network. ...
user avatar
0 votes
0 answers
21 views

Multioutput Neural Network for function approximation

I am trying to extend the example here to be capable of handling multiple outputs for function approximations ...
user avatar
0 votes
1 answer
15 views

Activation Functions in Haykins Neural Networks a comprehensive foundation

In Haykins Neural Network a comprehensive foundation, the piecwise-linear funtion is one of the described activation functions. It is described with: The corresponding shown plot is I don't really ...
user avatar
0 votes
1 answer
26 views

Why does using tanh worsen accuracy so much?

I was testing how different hyperparameters would change the output of my multilayer perceptron for a regression problem ...
user avatar
3 votes
4 answers
1k views

Neural Network not Deep

I have found this image link I would like to know what NNs are not deep neural? The first three? Also what kind of functional activations do they use?
user avatar
  • 459
0 votes
0 answers
31 views

Why using the hyperbolic tangent or the sigmoid as activation function on the last layer gaves the same result in accuracy?

The problem I'm making a simple Multilayer Perceptron (MLP), in Keras, that has to do the binary classification from some float type of data. Each single data is a group of three float values (e.g. 32....
user avatar
  • 1
0 votes
0 answers
10 views

Activation maps positiv even bevore activation

I was looking at the activation maps of vgg19 in pytorch. I found that all the values of the maps are positive even before I applied the ReLU. This seems very strange to me... If this would be correct ...
user avatar
1 vote
1 answer
70 views

How to prove Softmax Numerical Stability?

I was playing around with the softmax function and tried around with the numerical stability of softmax. If we increase the exponent in the numerator and denominator with the same value, the output of ...
user avatar
0 votes
0 answers
14 views

What happens if you don't include any activation function on hidden classification layers?

What happens if we don't apply an activation function to the classification hidden layers and apply it only for the final output layer (Sigmoid, Softmax)? I'm asking this because I have trained a CNN ...
user avatar
0 votes
2 answers
37 views

Is there any point in having layers in a neural network for regression problems?

In my textbook I read that an MLP and linear activation functions for the hidden layers can be reduced to a simple input-output system, i.e. no hidden layers. This makes sense to me. Later on I read ...
user avatar
0 votes
0 answers
11 views

Feature importance by removing all other features?

For neural network feature importance, can I zero-out all features except one in order to gauge that feature's importance? I know shuffling a feature is one approach. For example, leaving in the 4th ...
user avatar
  • 476
0 votes
2 answers
47 views

Activation and Loss Function not chosen correctly when use Neural Network

I have three classes for my text dataset before. These are my classes: 0 = Cat 1 = Not Both 2 = Dog Then I use this code: ...
user avatar
  • 13
2 votes
0 answers
16 views

Derive backpropagation for PreLU

I want to derive the back propagation functions for the Parametric Relu activation function which is defined as follows: $$ h_a(x) = \text{max}(ax, x) $$ I want to derive $ \frac{\partial L}{\partial ...
user avatar
  • 21
1 vote
0 answers
24 views

What meaning does exp function have? [closed]

Is there any problem that solution(or algorithm) would be exp function? Let's say f(x)=2^x. It's an answer of a problem when you would like to know how many pieces would be made when you fold a paper ...
user avatar
  • 21
1 vote
0 answers
22 views

The case of (1,478) dim and parameters of neural network to find out

colleagues, actually I am kind'a new to NN, but hard trying.. I have data: Index: 40073 entries (excluded from training, UID) Columns: 484 entries dtypes: bool(468), float64(2), int64(13), object(1) I ...
user avatar
7 votes
2 answers
692 views

Why deep learning models still use RELU instead of SELU, as their activation function?

I am a trying to understand the SELU activation function and I was wondering why deep learning practitioners keep using RELU, with all its issues, instead of SELU, which enables a neural network to ...
user avatar
1 vote
1 answer
4k views

Difference between ReLU, ELU and Leaky ReLU. Their pros and cons majorly

I am unable to understand when to use ReLU, ELU and Leaky ReLU. How do they compare to other activation functions(like the sigmoid and the tanh) and their pros and cons.
user avatar
1 vote
0 answers
60 views

Word embedding autoencoder

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated. Here is my ...
user avatar
0 votes
0 answers
26 views

Bounded regression problem: sigmoid, hard sigmoid or…?

I have been training a neural network for a bounded regression and I am still in doubt for which activation function to use on the output layer. At first, I was convinced that a sigmoid would be the ...
user avatar
1 vote
1 answer
72 views

Activation Function

I am very new to machine learning and made an experiment myself. I have a few questions: Can I use $Y = sin(x)$ or $Y = 2x$ as an activation function for a neural network? Is it necessary to increase ...
user avatar
  • 13
1 vote
2 answers
67 views

How can the ReLU function lead to convergence?

The gradient descent algorithm is based on the fact that the gradient decreases as we move towards the optimum point. However, in the activations by the ReLU ...
user avatar
1 vote
0 answers
44 views

If the input to the autoencoder is normalized, do we need to use sigmoid on the last layer?

According to: https://stackoverflow.com/questions/65307833/why-is-the-decoder-in-an-autoencoder-uses-a-sigmoid-on-the-last-layer The last layer activation function contains sigmoid in order to the ...
user avatar
1 vote
1 answer
78 views

Is there comprehensive list of activation functions and their applications for a Neural Network?

I am aware of common activation functions like sigmoid, tanh,ReLu, Leaky ReLu. Even heard about a function called Swish. Now is there any detailed information on other activations functions and some ...
user avatar
1 vote
1 answer
54 views

Can this dataset be separated linearly?

Is this dataset linearly separable? If not, can it be converted into one by applying some function as it seems to follow the same pattern? Also, which classification algorithms could be used to fit ...
user avatar
  • 13
2 votes
1 answer
2k views

How batch normalization layer resolve the vanishing gradient problem?

According to this article: https://towardsdatascience.com/the-vanishing-gradient-problem-69bf08b15484 The vanishing gradient problem occurs when using the sigmoid ...
user avatar
0 votes
1 answer
154 views

Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
user avatar
  • 277
1 vote
1 answer
41 views

Training set Distribution and Activation function/Loss function correlation

How should the probability distribution of the training set influence the choice of the activation function / loss function? For instance if I have a Multinoulli distribution, which activation ...
user avatar
2 votes
1 answer
46 views

Do Activation Functions map to Higher Dimensions?:

I just started learning tensorflow and I have a question regarding activation functions used in neural networks, I watched a 3b1b video a while ago and it seems it squished the value into an interval ...
user avatar
0 votes
1 answer
89 views

Numerically stable hyperbolic tangent

The hyperbolic tangent is commonly used as an activation function: $$ tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} $$ Although, it is unclear how this function is implemented to be numerically stable ...
user avatar
  • 153
0 votes
1 answer
78 views

Applying activation on part of the layer in Keras

Context I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network: ...
user avatar
0 votes
1 answer
47 views

Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...
user avatar
  • 103
0 votes
1 answer
78 views

Input and output layer activation functions of neurons in Orange

What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?
user avatar
  • 1
0 votes
1 answer
19 views

Can there be in-active neuron in output layer

I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a ...
user avatar
1 vote
0 answers
18 views

Does the number of hidden layers affect the activation function?

Suppose there's a network with N hidden layers. There are 2 cases: The network is deep The network is shallow I've been wondering how ...
user avatar
  • 11
0 votes
0 answers
56 views

tanh function values are either 1 or -1, how to interpret that distribution

I have a question regarding the tanh function. I trained an NN (with tanh activation functions in hidden layers) on a multiclass dataset and visualised the tanh values of the complete samples from the ...
user avatar
  • 73
1 vote
1 answer
27 views

Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
user avatar
  • 476
0 votes
1 answer
17 views

Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?

I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, ...
user avatar
  • 103
1 vote
0 answers
13 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
user avatar
0 votes
1 answer
73 views

How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?

I was looking into the multi-label classification on the output layer of a Neural Network. I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons. So for example an ...
user avatar
1 vote
1 answer
226 views

Scaling the activation function

It is obvious that I have to scale the output data if the range of values is between say [-10;10] and the activation function of the output layer takes values in the interval [-1;1]. But I could also ...
user avatar
  • 13
1 vote
1 answer
381 views

Problem with convergence of ReLu in MLP

I created neural network from scratch in python using only numpy and I'm playing with different activation functions. What I observed is quite weird and I would love to understand why this happens. ...
user avatar
1 vote
1 answer
209 views

Dying leaky ReLU

I am trying to train a deep neural network but I am having dying ReLU problem. I am using leaky Relu but still have the same problem. Isn't leaky relu supposed to not have such problems?
user avatar
1 vote
1 answer
111 views

Is it possible to get an ROC curve using Relu activation?

Based on my understanding, given that Relu doesn't provide probabilities unlike Softmax, it's not possible to plot an ROC curve. However, is there some way to convert the output from a Relu to ...
user avatar
  • 113
1 vote
0 answers
421 views

What is the reason behind Keras choice of default (recurrent) activation functions in LSTM networks

Activation function between LSTM layers In the above link, the answer to the question whether activation function are required for LSTM layers was answered as follows: as an LSTM unit already consists ...
user avatar
3 votes
1 answer
302 views

Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
user avatar
2 votes
1 answer
826 views

As RELU is not differentiable when it touches the x-axis, doesn't it effect training?

When I read about activation functions , I read that the reason we don't use step function is because, it is non differentiable which leads to problem in gradient descent. I am a beginner in deep ...
user avatar
  • 669
2 votes
1 answer
154 views

Output landscape of ReLU, Swish and Mish

I found the following figure in the original Mish paper (https://arxiv.org/abs/1908.08681). I understand that this figure describes how the loss is being changed, if the change is smooth or not. But ...
user avatar
1 vote
0 answers
27 views

Sharing parameters of an activation across layers of a neural network

Keras now provides advanced parametric activation layers like Leaky-ReLU PReLU. Each time I add this layer to a sequential model, an additional trainable parameter is added to graph. How can I make ...
user avatar
  • 163