Questions tagged [activation-function]

Activation function is a non-linear transformation, usually applied in neural networks to the output of the linear or convolutional layer. Common activation functions: sigmoid, tanh, ReLU, etc.

Filter by
Sorted by
Tagged with
2
votes
2answers
894 views

Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
2
votes
1answer
1k views

Implementing a custom hard sigmoid function

I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values: 0 if x < 0 1 if x > 1 x if 0 <= x <= 1 How do I implement it with a ...
1
vote
1answer
457 views

The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type ...
8
votes
4answers
15k views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
2
votes
1answer
52 views

“Each agent was evaluated every 250,000 training frames for 135,000 validation frames” What does this sentences stands for? in DQN nature paper?

In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions? 0- What is the meaning of this ...
24
votes
4answers
37k views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How do you use LeakyRelu as an activation function in sequence DNN in keras? If I want to write something similar to: ...
1
vote
1answer
320 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
2
votes
1answer
31 views

Restricting the output of a model didn't improve the loss value of the model evaluation

There is a deep model for prediction. The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80) The model Loss value is 70 and I would like to reduce it. I printed the ...
1
vote
1answer
3k views

Neural network example not working with sigmoid activation function

I'm running the Neural Network example written in in BogoToBogo The program worked fine: ...
1
vote
1answer
5k views

Why is the softmax function often used as activation function of output layer in classification neural networks?

What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
3
votes
1answer
444 views

How can ReLU ever fit the curve of x²?

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations: Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ ...
1
vote
1answer
51 views

How to display the value of activation?

I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "...
3
votes
4answers
87 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
0answers
1k views

Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...
6
votes
3answers
7k views

Activation function vs Squashing function

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer. Today, in a video explaining deep neural networks, I came across the term Squashing ...
0
votes
1answer
400 views

Properly using activation functions of neural network

I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear. Input Layer: From my ...
2
votes
2answers
2k views

Alternatives to linear activation function in regression tasks to limit the output

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
2
votes
0answers
82 views

Cross error loss function cause division by zero error

How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by: -(ylog(actual_output)+(1-y)*(1-...
1
vote
2answers
3k views

Is it possible to customize the activation function in scikit-learn's MLPRegressor?

Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway. Is it possible to change the activation function of the output layer in an MLPRegressor neural ...
1
vote
2answers
105 views

Weights in neural network

So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data ...
1
vote
2answers
114 views

Homemade deep learning library: numerical issue with relu activation

For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade. It seems to work ...
0
votes
1answer
292 views

What exactly is the “hyperbolic” tanh function used in the context of activation functions?

I know the plot of $\tanh$ activation function looks like. I also know that its output has a range of $[-1, 1]$. Furthermore, I also know the it is defined as follows $$ \tanh(x) = \frac{\sinh(x)}{...
1
vote
0answers
38 views

How to Implement Biological Neuron Activations in Artificial Neural Networks

In artificial neural networks, activation functions are used for neurons, i.e. the sigmoid activation: Which can be implemented in code as (in Python): ...
3
votes
2answers
135 views

Is classifier able to say there's no-such-case?

I am a starter in ML and I need some help... The problem Assume that I have a classifier which can classify left hand / right hand well. I am curious whether it can decide whether there's a hand in ...
37
votes
3answers
22k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
2
votes
1answer
3k views

Negative Rewards and Activation Functions

I have a question regarding appropriate activation functions with environments that have both positive and negative rewards. In reinforcement learning, our output, I believe, should be the expected ...
4
votes
1answer
791 views

Is there a way to set a different activation function for each hidden unit in one layer in keras?

I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
9
votes
1answer
10k views

Input normalization for ReLu?

Let's assume a vanilla MLP for classification with a given activation function for hidden layers. I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
5
votes
2answers
773 views

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks? Do they perform better than non-monotonic ones? Is this mathematically proven? Are ...
-1
votes
1answer
7k views

Error in f1(x) : argument “b” is missing, with no default

f1 <- function(a,b,c,d,e,f) { -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f } ...
1
vote
0answers
116 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
3
votes
1answer
196 views

Why is an activation function notated as “g”?

In many cases an activation function is notated as g (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as ...
18
votes
1answer
31k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
5
votes
1answer
252 views

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
19
votes
1answer
17k views

Difference of Activation Functions in Neural Networks in general

I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear. It's reasonable that one ...
7
votes
2answers
4k views

What is the purpose of multiple neurons in a hidden layer?

On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer. Let me make the question more clear. Take this ...
3
votes
3answers
4k views

Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects: simple more biologically plausible no gradient to vanish better performance sparsity And I see only one ...

1 2
3