2
$\begingroup$

I understand that gradient descent is local and it deals only with the inputs to the neuron, what it outputs and what it should output. In all I've seen, gradient descent needs the activation function to be differentiable, so a threshold function cannot be used.

Yet, the biological neurons either fire or they don't. The input to the neuron, in my understanding, is the equivalent of the membrane potential. Once it passes a certain threshold, the neuron fires (one or multiple times) and the input is reset. Considering this, the step function appears to be enough to reproduce the behavior of biological neurons.

I'm thinking about the usual weights (integer or even float) paired with the step function, trained with the following form of backpropagation:

  • looking at one neuron and its connections and backpropagating, it has an output of 1 (it fired) when it should have been 0. The neuron has a threshold of 5 and its net input did sum up to be higher than 5.

  • to train it, subtract from the weights so that the net input decreases by an arbitrary learning rate, so that the threshold is not exceeded or is closer to not being exceeded. For example, the net input was 5.5, so the "error" is 0.5. Multiply this "error" by the learning rate of 0.1 and the result is 0.05. This means that the weights need to be adjusted by 5% (multiplied times 0.95). The weights are changed only a little for each training example.

In my understanding, this is similar to backpropagation with gradient descent, but it's applied to step activation functions and it works even if they're not differentiable.

My question is, where are the implementations of this very simple training method? Why wouldn't it work?

$\endgroup$
2
  • $\begingroup$ For a hidden layer how would you know that an output should be 0 or 1? $\endgroup$
    – JahKnows
    Feb 21, 2018 at 15:23
  • $\begingroup$ Could be done like in gradient descent, by summing up the errors of the units that follow it. If all the neurons that follow need a smaller net input, clearly, the preceding neuron must not fire. $\endgroup$
    – user196060
    Feb 21, 2018 at 17:26

1 Answer 1

0
$\begingroup$

There are not common implementations of the proposed method because there is not much to be gained by adding a step function to a neural network. A step function increases the complexity without significantly increasing the learning capacity of neural networks beyond a nonlinear activation function.

The goal of neural networks is not to model biological neurons. The goal of spiking neuron models is to model biological neurons. Spiking neuron models often use the functions you describe.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.