I am totally new to Artificial Neural Networks. Let’s say that the model you are trying to turn into an artificial neural network has an output that is triggered only by the exceedance of a threshold: $y\geq y_{1}. $Therefore, you need to find a way to use this inequality as an activation function. Is this feasible?

  • $\begingroup$ Do you know $y_1$ or is that something to be learned? $\endgroup$ – Jan van der Vegt May 22 '17 at 14:07
  • $\begingroup$ @JanvanderVegt I already know y1. Shall I use a Unit Step function as an activation function? I am learning something in the process. $\endgroup$ – FaCoffee May 22 '17 at 14:12
  • $\begingroup$ A problem with doing that is that derivates are not able to flow through that anymore. But I don't think I fully understand the issue, if you know $y_1$ already, can't you transform your input? Can you give an example usecase? $\endgroup$ – Jan van der Vegt May 22 '17 at 14:33
  • $\begingroup$ Yes, I can do that. But then I don't need an activation function anymore, because the inequality would result in a binary case: it's either 0 or 1. In that case, would that still be a neural network? $\endgroup$ – FaCoffee May 22 '17 at 15:02
  • 1
    $\begingroup$ Can you give an example use case? If you have a binary switch then you cannot train the network properly because of not having proper subgradients flowing with backpropagation $\endgroup$ – Jan van der Vegt May 22 '17 at 15:06

This is feasable. This is also called a Binary/Step activation function. You must only use this activation function on the output neurons.

The Step function will round down an answer that is lower than 0.5 to 0, and an answer that is higher than 0.5 to 1. However, please note that you do not need to use a binary activation function to output 1 - I advise you to just use TanH or sigmoid and backpropagate a whole bunch of iterations.

However, in other comments, you mentioned that you know what y1 is. That is not of importance, the network will act as a black box and will figure out a treshold itself. Don't set up your own activation function just to get the right output - that avoids the whole point of backpropagation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.