# Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

169 questions
Filter by
Sorted by
Tagged with
1 vote
10 views

### What is a proper activation function with simulated-annealing trainer for neural network?

I'm developing a gpu-accelerated simulated annealing based neural network trainer library. Currently its stuck on how to converge on "array sorting by neural network 3:10:20:10:3 topology". ...
1 vote
31 views

### Why would multiple activation layers be used in a row?

I'm learning about ML and I was looking at ...
42 views

### Alternative to ELU and Leaky ReLU?

I was talking with a friend about different activation functions (we are still new to ML). One thing that I didn't like about ELU was the vanishing gradient, and about Leaky ReLU that it's not ...
19 views

### Relu derivative value

I have a stupid question on the derivative of relu activation function. After the finding the difference of the true output $t_k$ and predicted output $a_k$, why is the value of the $d_{a3}$ \ $d_{z3}$...
25 views

### Query about Sigmoid activation function calculation

While applying sigmoid activation function (in finding y label), I have calculated it as below: y = 0.35 + (0.8 * 0.1) + (0.3 * 0.6) + (-0.2 * 0.4) = 0.53 sigmoid_y = 0.625 how do we take threshold ...
43 views

### How to implement a custom loss with a non-mathematical operation (simulation) that backpropagates with PyTorch?

I am writing a Neural Network, which output is not used directly for the loss-function, but rather as the input for a simulation model. After the simulation ran, I am using the simulated_value and the ...
104 views

### How ReLU is bringing non linearity and why it is not an alternative to dropout?

The differentiation of ReLU function is 1 when input is greater than 0, and 0, when input is less than or equal to 0. In the backpropagation process it doesn’t change the value of d(error)/d(weight) ...
255 views

### Why use tanh (or any other activation function)?

In machine learning, it is common to use activation functions like tanh, sigmoid, or ReLU to introduce non-linearity into a neural network. These non-linearities help the network learn complex ...
124 views

### Regression model doesn't handle negative values

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation. To do this, I'm using the ReLU activation function for each layer. The ...
16 views

193 views

### Applying activation on part of the layer in Keras

Context I am trying to implement the YOLO algorithm in Keras. What I have so far is the following network: ...
80 views

### Is it possible to implement a vectorized version of a Maxout activation function?

I want to implement an efficient and vectorized Maxout activation function using python numpy. Here is the paper in which "Maxout Network" was introduced (by Goodfellow et al). For example, ...