Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
18
votes
1answer
31k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
35
votes
3answers
22k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
7
votes
2answers
4k views

What is the purpose of multiple neurons in a hidden layer?

On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer. Let me make the question more clear. Take this ...
18
votes
1answer
17k views

Difference of Activation Functions in Neural Networks in general

I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear. It's reasonable that one ...
8
votes
4answers
15k views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
3
votes
2answers
2k views

ReLU for combating the problem of vanishing gradient in RNN?

For solving the problem of vanishing gradients in feedforward neural networks, ReLU activation function can be used. When we talk about solving the vanishing gradient problem in RNN, we use a more ...
2
votes
2answers
2k views

Alternatives to linear activation function in regression tasks to limit the output

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
1
vote
0answers
27 views

Which activation function of the output layer and which loss function are advised to be used for bounded regression?

I want my (deep) neural network to produce an output from a certain range, in my case between 0 and 255. I have scaled the labels from [0..255] to [0..1]. For the neural network, I have tried a ...