Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [activation-function]

The tag has no usage guidance.

0
votes
0answers
5 views

Obtaining correctly gradient in neural network of output with respect to input. Is relu a bad option as the activation function?

My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus. My main purpose is to ...
0
votes
1answer
12 views

What does it mean for an activation function to be “saturated/non-saturated”?

For context, in this paper Several RNN variants such as the long short-term memory (LSTM) [10, 18] and the gated recurrent unit (GRU) [5] have been proposed to address the gradient problems. ...
0
votes
0answers
9 views

Are ReLUs used for training deep Boltzmann machines (DBMs) in an unsupervised setting?

Sigmoid activations follow naturally from the definition of Boltzmann distributions. I understand that for supervised tasks, ReLUs are now commonly used because of better performance/no exploding/...
1
vote
0answers
21 views

Why activation function is not needed during the runtime of an Word2Vec model

In Word2Vec trainable model, there are two different weight matrix. The matrix $W$ from input-to-hidden layer and the matrix $W'$ from hidden-to-output layer. Referring to this article, I understand ...
2
votes
1answer
34 views

Implementing a custom hard sigmoid function

I need to implement an activation function that is similar to Keras's "hard-sigmoid", only for different limit values: 0 if x < 0 1 if x > 1 x if 0 <= x <= 1 How do I implement it with a ...
1
vote
1answer
26 views

The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

First of all, I know the usage of leaky RELUs and some other relevant leaky activation functions as well. However I have seen in a lot of papers on object detection tasks (e.g YOLO) to use this type ...
0
votes
0answers
12 views

Adaptive Activation functions using Keras

I am new to Neural Nets. I am trying to use the activation function mentioned in this research paper, https://arxiv.org/pdf/1412.6830.pdf I am trying to use keras and I don't understand how to input ...
0
votes
0answers
35 views

Which loss and activation function at output layer is suitable for Multi target classification problem?

I am modelling a Multi-target classification problem which has 220 input features and 132 output features. Each output target has an integer value in between [0,1,2,3,4] .And for this I have applied ...
0
votes
0answers
19 views

normalize output of a feed-forward ANN

My feed-forward neural-network is modeling (regression) a multi-channel loss function. The output of the network is a vector y (size 10) that describes the loss ratio of the input signal x for each ...
2
votes
2answers
247 views

How does Sigmoid activation work in multi-class classification problems

I know that for a problem with multiple classes we usually use softmax, but can we also use sigmoid? I have tried to implement digit classification with sigmoid at the output layer, it works. What I ...
2
votes
1answer
45 views

“Each agent was evaluated every 250,000 training frames for 135,000 validation frames” What does this sentences stands for? in DQN nature paper?

In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions? 0- What is the meaning of this ...
1
vote
1answer
899 views

How to use LeakyRelu as activation function in sequence DNN in keras?When it perfoms better than Relu?

How to use LeakyRelu as activation function in sequence DNN in keras? if I want to write something similar to: model = Sequential() model.add(Dense(90, activation='LeakyRelu')) What is the ...
1
vote
1answer
28 views

best activation function for ensemble?

i have created some logistic regression model (different preprocessing) with softmax function. and i mix all model with an ensemble with a hierarchical method. so the output of all model (base) will ...
2
votes
1answer
18 views

Restricting the output of a model didn't improve the loss value of the model evaluation

There is a deep model for prediction. The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80) The model Loss value is 70 and I would like to reduce it. I printed the ...
1
vote
1answer
183 views

Neural network example not working with sigmoid activation function

I'm running the Neural Network example written in in BogoToBogo The program worked fine: ...
1
vote
1answer
399 views

Why is the softmax function often used as activation function of output layer in classification neural networks?

What special characteristics of the softmax function makes it a favourite choice as activation function in output layer of classification neural networks?
1
vote
0answers
18 views

Rules of thumb relating input range of values to choice of activation function

I would like to check with the experts on some observations I made about input value range and choice of activation function in deep learning neural networks. Here are some rules of thumbs I have; ...
3
votes
1answer
63 views

How can ReLU ever fit the curve of x²?

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations: Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ ...
1
vote
1answer
27 views

How to display the value of activation?

I have built my network and would like to see how the activation of a particular layer change after each epoch of training. For example, as code shown below, I want to see the activation values of "...
1
vote
2answers
28 views

Few activation functions handling various problems - neural networks

How can a few activation functions in neural networks handle so many different problems? I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in ...
1
vote
0answers
124 views

Understanding of threshold value in a neural network

Consider the hypothetical neural network here $o_1$ is the output of neuron 1. $o_2$ is the output of neuron 2. $w_1$ is the weight of connection between 1 and 3. $w_2$ is the weight of connection ...
3
votes
2answers
435 views

Activation function vs Squashing function

This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer. Today in a video explaining deep neural networks I came across the term Squashing ...
0
votes
0answers
24 views

Why is dropout a good fit for ReLU units?

I have read this before that dropout interacts well with ReLU and recently going through SNN paper i came acoss this again. to quote: Dropout fits well to rectified linear units, since zero is in the ...
0
votes
1answer
83 views

Properly using activation functions of neural network

I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear. Input Layer: From my ...
0
votes
0answers
168 views

Why do we use leakyRELU over RELU in Variational Autoencoders?

I have seen in multiple research papers and algorithmic implementations where leakyRELU is used for variational auto encoders. Knowing the fact that leakyRELU is used to obtain non-zero gradient for ...
1
vote
1answer
269 views

Alternatives to linear activation function in regression tasks to limit the output

I want to know whether there is a way to limit the output of a regression deep model. Suppose that I want my model outputs values which are in a specified range and penalizes the outputs which are not ...
2
votes
0answers
56 views

Cross error loss function cause division by zero error

How to calculate cross entropy when actual output is 0? Would not it give indf brcause of log(0) and the cross entropy for binary classification is given by: -(ylog(actual_output)+(1-y)*(1-...
1
vote
1answer
503 views

Is it possible to customize the activation function in scikit-learn's MLPRegressor?

Similar to this question about MLPClassifier, I suspect the answer is 'no' but I will ask it anyway. Is it possible to change the activation function of the output layer in an MLPRegressor neural ...
0
votes
0answers
30 views

Restricting the output of a neural network

I am training a neural network to predict the age of a person from an image. I am treating this as a regression problem. I want to restrict the output of the neural network from 0 to 120. How can I do ...
1
vote
2answers
49 views

Weights in neural network

So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data ...
1
vote
2answers
62 views

Homemade deep learning library: numerical issue with relu activation

For the sake of learning the finer details of a deep learning neural network, I have coded my own library with everything (optimizer, layers, activations, cost function) homemade. It seems to work ...
0
votes
1answer
122 views

What exactly is the “hyperbolic” tanh function used in the context of activation functions?

Below are the information I know about tanh I know the plot of tanh activation function and that its output has a range of [-1 1]. I also know the equation My Questions What exactly is the meaning ...
1
vote
0answers
34 views

How to Implement Biological Neuron Activations in Artificial Neural Networks

In artificial neural networks, activation functions are used for neurons, i.e. the sigmoid activation: Which can be implemented in code as (in Python): ...
0
votes
0answers
78 views

Generative Adverserial Networks - SELU instead of RELU

I recently read a paper on Hight-Resolution Deep Convolutional Generative Adverserial Networks, or HDCGANs for short. And in the paper they have been using SELU instead of RELU. I gone through it ...
3
votes
2answers
113 views

Is classifier able to say there's no-such-case?

I am a starter in ML and I need some help... The problem Assume that I have a classifier which can classify left hand / right hand well. I am curious whether it can decide whether there's a hand in ...
13
votes
2answers
4k views

Why is ReLU used as an activation function?

Activation functions are used to introduce non-linearities in the linear output of the type w * x + b in a neural network. Which I am able to understand ...
0
votes
0answers
426 views

Why many networks (inception-resnet-v2, inception v4/v3) do not use activation function on the last layer?

I was looking into the source code of tensorflow models: https://github.com/tensorflow/models/tree/master/research/slim/nets And many of the does not use activation function on the last layer (logits)...
1
vote
1answer
1k views

Negative Rewards and Activation Functions

I have a question regarding appropriate activation functions with environments that have both positive and negative rewards. In reinforcement learning, our output, I believe, should be the expected ...
3
votes
1answer
316 views

Is there a way to set a different activation function for each hidden unit in one layer in keras?

I'm trying to set a different activation function for each hidden unit in a layer. Is this possible in Keras with 'Concatenate'?
3
votes
1answer
3k views

Input normalization for ReLu?

Let's assume a vanilla MLP for classification with a given activation function for hidden layers. I know it is a known best practice to normalize the input of the network between 0 and 1 if sigmoid ...
4
votes
2answers
261 views

Advantages of monotonic activation functions over non-monotonic functions in neural networks?

What are the advantages of using monotonic activation functions over non-monotonic functions in neural networks? Do they perform better than non-monotonic ones? Is this mathematically proven? Are ...
-1
votes
1answer
3k views

Error in f1(x) : argument “b” is missing, with no default

f1 <- function(a,b,c,d,e,f) { -111.605*a-208.39+(14.882-b)^2+35.29813*c-.001251205/d-1.050695*e+11.63420*f } ...
1
vote
0answers
63 views

Why isn't Maxout used in the state of the art models?

I have just read the paper from Ian Goodfellow et al. titled "Maxout Networks". It seems that the Maxout activation should be quite powerful, as it can approximate any convex function, i.e. Relu, ...
1
vote
1answer
97 views

Why is an activation function notated as “g”?

In many cases an activation function is notated as g (e.g. Andrew Ng's Course courses), especially if it doesn't refer to any specific activation function such as ...
9
votes
1answer
10k views

Why ReLU is better than the other activation functions

Here the answer refers to vanishing and exploding gradients that has been in sigmoid-like activation functions but, I guess, Relu...
4
votes
1answer
178 views

Why do so many functions used in data science have derivatives of the form f(x)*(1-f(x))?

The sigmoid function's derivative is of that form, and so is the softmax function's. Is this by design, or some strange coincidence that seems to work for ML models/neural networks?
14
votes
1answer
11k views

Difference of Activation Functions in Neural Networks in general

I have studied the activation function types for neural networks. The functions themselves are quite straightforward, but the application difference is not entirely clear. It's reasonable that one ...