# Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

139 questions
Filter by
Sorted by
Tagged with
2k views

### What are best activation and regularization method for LSTM?

In Keras there are: ...
48 views

### Regression with -1,1 target range - Should we use a tanh activation in the last 1 unit dense layer?

Say in a regression problem the target range to be between [0,1] or [-1,1], and say the last layer of the network is as ...
480 views

### The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type ...
324 views

### best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
1k views

### Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...
41 views

### How to Implement Biological Neuron Activations in Artificial Neural Networks

In artificial neural networks, activation functions are used for neurons, i.e. the sigmoid activation: Which can be implemented in code as (in Python): ...
116 views

### Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
488 views

### How does cost function change by choice of activation function (ReLU, Sigmoid, Softmax)?

I am new to ML and as I take courses for the area DL, I am wondering, by our choice of activation function for the last layer, whether we take sigmoid, relu or softmax, would the formula for ...
406 views

### Properly using activation functions of neural network

I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear. Input Layer: From my ...
39 views

### Numerically stable hyperbolic tangent

The hyperbolic tangent is commonly used as an activation function: $$tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$ Although, it is unclear how this function is implemented to be numerically stable ...
93 views

### What is the meaning of colors in activation maps?

I am trying to understand how the neural network works. I plotted the intermediate layer activations and I got the following image. I used the matplotlib library and selected ...
87 views

### Is there a limit in the number of layers for neural network?

I heard the neural network has a problem with vanishing gradient problems even though the ReLU activation function is used. In ResNet(that has a connecting function for reducing the problem), there ...
3k views

### Why there is no exact picture of softmax activation function?

I was wondering why there is no precise picture of the softmax activation function on the internet. Is it difficult for the plot or what is the reason behind that since I want to compare it with a ...
33 views

### Deep Neural Network and Activation Function

I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in, Increasing model complexity and introduce non-linearity Avoiding Gradient Vanishing ...
295 views

### What exactly is the "hyperbolic" tanh function used in the context of activation functions?

I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows  \tanh(x) = \frac{\sinh(x)}{...
27 views

### Why the gradient of a ReLU for X>0 is 1?

Gradient is derivative of several variables. I can't understand why is the gradient of a ReLU for X>0 is 1 ? and 0 for x < 0 ? I tried to search for proof and examples but didn't found any good ...
13 views

### Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data?

I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, ...
68 views

### In backpropagation, scale is also important?

I think backpropagation is needed to find the direction of gradient decent method. I also wonder, the scale is also important? I heard some issue of vanishing(or exploding) gradient problem. If the ...
187 views

### PELU activation: how does it work, and how to implement?

I have encountered PELU (Parametric Exponential Linear Unit) in the literature, but I can't find practical applications of it. Moreover, I have some questions about how it works: Are its parameters ...
231 views

### Modulo as activation function in neural network

Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our ...
74 views

### Quasi-linearity in deep learning regression problems (sports betting)

I’m attempting to build a sports betting model that aims to predict final scores for games. I’ve had some promising early results for US college football just by using linear regression to form team ...
10 views

### Bounded regression problem: sigmoid, hard sigmoid or…?

I have been training a neural network for a bounded regression and I am still in doubt for which activation function to use on the output layer. At first, I was convinced that a sigmoid would be the ...
29 views

### Applying activation on part of the layer in Keras

Context I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network: ...
36 views

### Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...
34 views

### Input and output layer activation functions of neurons in Orange

What are the activation functions of the neurons in the input and output layer of a neural network model from the Orange machine learning application?
16 views

### Can there be in-active neuron in output layer

I am new to deep learning, and was studying about it. I know that input from input layer is multiplied with weights and then added with bias. And output of this is passed to a activation function of a ...
14 views

### Guidelines to efficiently train neural networks which have polynomial activations

I am interesting in studying low footprint NN by replacing the activation functions with low-degree polynomial approximations. Doing this, I am fine with slight reduction in accuracy if I the training ...
28 views

### tanh function values are either 1 or -1, how to interpret that distribution

I have a question regarding the tanh function. I trained an NN (with tanh activation functions in hidden layers) on a multiclass dataset and visualised the tanh values of the complete samples from the ...
19 views

### Sigmoid vs Softmax with cross entropy

For a binary classifier, Is there any difference in Sigmoid activation function vs Softmax with cross entropy ? In this answer how-to-maximize-recall it is stated that if we use softmax with cross ...
19 views

### Sigmoid activation function for scaled continuous data

I've been working on a NLP project that attempt to output a single numeric value. The natural form of the data is integers between 0 and 27, with 27 being an absolute maximum, and values above 27 ...
9 views

### How to determine activation functions for neural network

I am trying to plan a neural network for regression predictions. The final activation layer should be a linear function, but for hidden layers, do the activation functions need to also be all linear ...
41 views

### How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?

I was looking into the multi-label classification on the output layer of a Neural Network. I have 5 Output Neurons where each Neuron can be 1, 0, or -1. independent of other Neurons. So for example an ...
90 views

### Avoid saturation in activation function (sigmoid)

Background It is said Sigmoid/Tanh are not to use because of the saturation issue. Question The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -...
27 views

### proper activation function at output and loss function to optimize for OCR?

I am trying to make a CNN model on IAM handwritten words data(which has images of words handwritten by multiple people and targets are text in the images). So, I can encode words to numbers(A=0, B=1 ...
68 views

### Example of convex activation function

For a particular task, I need a convex activation function with the following properties: f''(x) > 0 0 <= f(x) <= 1 f(x) is monotonic f(x) is not "exploding" i.e. avoiding functions such as ...
105 views

### Is it reasonable to use the output of the sigmoid function as the win rate prediction?

I'm working on a project which is predicting the win rate of one team or one person. (could be any kind of sports like baseball, basketball or e-sport games) The data I have is more like a ...
21 views

### What are contexts where a perceptron model could be defined?

per this post, a perceptron could use the logit function as the activation function. per wiki In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step ...