Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
3
votes
1answer
677 views

Is there a way to set a different activation function for each hidden unit in one layer in keras?

I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
6
votes
1answer
9k views

Input normalization for ReLu?

Let's assume a vanilla MLP for classification with a given activation function for hidden layers. I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
5
votes
2answers
655 views

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks? Do they perform better than non-monotonic ones? Is this mathematically proven? Are ...
-1
votes
1answer
6k views

Error in f1(x) : argument “b” is missing, with no default

f1 <- function(a,b,c,d,e,f) { -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f } ...
1
vote
0answers
111 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
3
votes
1answer
159 views

Why is an activation function notated as “g”?

In many cases an activation function is notated as g (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as ...
17
votes
1answer
27k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
5
votes
1answer
231 views

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
18
votes
1answer
15k views

Difference of Activation Functions in Neural Networks in general

I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear. It's reasonable that one ...
7
votes
2answers
4k views

What is the purpose of multiple neurons in a hidden layer?

On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer. Let me make the question more clear. Take this ...
3
votes
3answers
4k views

Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects: simple more biologically plausible no gradient to vanish better performance sparsity And I see only one ...

1 2
3