# Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

45 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
163 views

### How does one derive the modified tanh activation proposed by LeCun?

In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form: $$f(x) = 1.7159 * tanh(\frac{2}{3}*x)$$ ...
1k views

### Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
124 views

### Why the sigmoid activation function results in sub-optimal gradient descent?

I need some help understanding the second shortcoming of the sigmoid activation function as described in this video from Stanford. She says that because the output of sigmoid is always positive, that ...
90 views

### Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
71 views

### Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus. My main purpose is to ...
951 views

### Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
82 views

### Cross error loss function cause division by zero error

How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by: -(ylog(actual_output)+(1-y)*(1-...
13 views

### The case of (1,478) dim and parameters of neural network to find out

colleagues, actually I am kind'a new to NN, but hard trying.. I have data: Index: 40073 entries (excluded from training, UID) Columns: 484 entries dtypes: bool(468), float64(2), int64(13), object(1) I ...
28 views

### Word embedding autoencoder

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated. Here is my ...
12 views

### If the input to the autoencoder is normalized, do we need to use sigmoid on the last layer?

According to: https://stackoverflow.com/questions/65307833/why-is-the-decoder-in-an-autoencoder-uses-a-sigmoid-on-the-last-layer The last layer activation function contains sigmoid in order to the ...
18 views

### Does the number of hidden layers affect the activation function?

Suppose there's a network with N hidden layers. There are 2 cases: The network is deep The network is shallow I've been wondering how ...
22 views

### Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
13 views

### Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
267 views

### What is the reason behind Keras choice of default (recurrent) activation functions in LSTM networks

Activation function between LSTM layers In the above link, the answer to the question whether activation function are required for LSTM layers was answered as follows: as an LSTM unit already consists ...
21 views

### Sharing parameters of an activation across layers of a neural network

Keras now provides advanced parametric activation layers like Leaky-ReLU PReLU. Each time I add this layer to a sequential model, an additional trainable parameter is added to graph. How can I make ...
22 views

### Using Iterative Hard/Soft Thresholding in autoencoder with non linear activation

Can someone please give an intuitive explanation of the difference between the Iterative Hard Thresholding VS Iterative Soft thresholding algorithm? And if we can use these algorithms in an ...
13 views

### Wich activation function for DQL

After many research, I still can't find a neat answer about this question: When I found the loss of my state-action pair. I'm only backpropagating that loss true the network and setting all other ...
114 views

### Should output data scaling correspond to the activation function's output?

I am building an LSTM with keras which have an activation parameter in the layer. I have read that scaling on the output data ...
191 views

### Generalized softmax derivative for implementation with any loss function

I am currently taking some deep learning and neural network (NN) courses, and in addition to performing the course work, am implementing my own "toolkit" of NN techniques to better my understanding of ...
31 views

### Which activation function of the output layer and which loss function are advised to be used for bounded regression?

I want my (deep) neural network to produce an output from a certain range, in my case between 0 and 255. I have scaled the labels from [0..255] to [0..1]. For the neural network, I have tried a ...
242 views

### Writing a piecewise linear function as a sum of ReLU functions

Suppose I have a piecewise linear function $f(x) = \sum^n_{i=1}a_i\phi_i(x)$, where $\{\phi_i\}_{i=1}^n$ is a finite dimension space of dimension $n-1$, in particular I am interested in the functions ...
18 views

### How does one use activation function with greater than [-1;1] range for binary classification?

In Efficient Backprop (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), Lecun and others propose to use activation function that don't reach target values on their asypmptotes. They explain (§ 4....
60 views

### Should I scale/normalize the data before training a feedforward neural network using only lagged values?

So I am trying to forecast the price of a certain commodity that is quite volatile. Now I wanted to train a regression feedforward neural network to predict the price of next week. Just to be clear, ...
267 views

### Relationship between Sigmoid and Gaussing Distribution

I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?": The assumption of a dependent variable to follow ...
869 views

### What are the best activation functions for Binary text classification in neural networks?

I know that there are many activation functions like Relu, sigmoid, tanh ..etc, I just want to know the best one for my case - Binary text classification. I have heard that Relu is best for Binary ...
3k views

### What are best activation and regularization method for LSTM?

In Keras there are: ...
49 views

### Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as ...
1k views

### Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...
42 views

### How to Implement Biological Neuron Activations in Artificial Neural Networks

In artificial neural networks, activation functions are used for neurons, i.e. the sigmoid activation: Which can be implemented in code as (in Python): ...
116 views

### Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
10 views

### Bounded regression problem: sigmoid, hard sigmoid or…?

I have been training a neural network for a bounded regression and I am still in doubt for which activation function to use on the output layer. At first, I was convinced that a sigmoid would be the ...
33 views

### Applying activation on part of the layer in Keras

Context I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network: ...
39 views

### Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...
36 views

### Input and output layer activation functions of neurons in Orange

What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?
16 views

### Can there be in-active neuron in output layer

I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a ...
14 views

### Guidelines to efficiently train neural networks which have polynomial activations

I am interesting in studying low footprint NN by replacing the activation functions with low-degree polynomial approximations. Doing this, I am fine with slight reduction in accuracy if I the training ...
29 views

### tanh function values are either 1 or -1, how to interpret that distribution

I have a question regarding the tanh function. I trained an NN (with tanh activation functions in hidden layers) on a multiclass dataset and visualised the tanh values of the complete samples from the ...
19 views

### Sigmoid vs Softmax with cross entropy

For a binary classifier, Is there any difference in Sigmoid activation function vs Softmax with cross entropy ? In this answer how-to-maximize-recall it is stated that if we use softmax with cross ...
19 views

### Sigmoid activation function for scaled continuous data

I've been working on a NLP project that attempt to output a single numeric value. The natural form of the data is integers between 0 and 27, with 27 being an absolute maximum, and values above 27 ...
9 views

### How to determine activation functions for neural network

I am trying to plan a neural network for regression predictions. The final activation layer should be a linear function, but for hidden layers, do the activation functions need to also be all linear ...
96 views

### Avoid saturation in activation function (sigmoid)

Background It is said Sigmoid/Tanh are not to use because of the saturation issue. Question The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -...
27 views

### proper activation function at output and loss function to optimize for OCR?

I am trying to make a CNN model on IAM handwritten words data(which has images of words handwritten by multiple people and targets are text in the images). So, I can encode words to numbers(A=0, B=1 ...
75 views

### Example of convex activation function

For a particular task, I need a convex activation function with the following properties: f''(x) > 0 0 <= f(x) <= 1 f(x) is monotonic f(x) is not "exploding" i.e. avoiding functions such as ...