I'm trying to visualize how two perceptrons converge to two different decision boundaries (which is ultimately used to create the classifier for the non-linearly separable data).

enter image description here Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron? I know the weights can be initialized differently but does this second perceptron classify something else? Shouldn't the decision boundaries be converge to be the the same ultimately?

enter image description here Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

  • 1
    $\begingroup$ Can you provide more information about the problem you look at (code, statement of the problem) $\endgroup$ – Robin Nicole Feb 28 '19 at 11:25
  • $\begingroup$ @RobinNicole Please have a look, I've added some diagrams and edited my question as well. $\endgroup$ – ashar Mar 1 '19 at 23:41

In short, because you are not training the individual hidden neurons, but the entire network.

Thinking holistically, your little net there is just a function with 9 parameters (two weights and a bias at each non-input node). Ignoring the structure of that function for now, you're just applying gradient descent (or maybe some variant thereof) to minimize some error function on your training set. There's no reason for this to converge to a point where the parameters at $p_0, p_1$ are the same.

When you compute the gradient, it's "at" $p_2$, so you're asking how to move the two decision boundaries (well, also the scaling of the scores on one side of the boundary, and how the two boundaries are combined into the final one) in order to improve the overall output best. Since the model will generally do better with a more complicated overall decision boundary, it will move the two hidden boundaries in different directions; it will not try to make them the same!

See also What is the purpose of multiple neurons in a hidden layer?


In the article which you linked, it seems that none of the two perceptrons are trained on data, you just have two perceptrons with different weights w0 and w1. If they have different weights, then they will have two different decision boundaries. You can see that, looking at the code that plots the decision boundaries:

ax.plot(t0, decision_boundary(w0, t0), 'm', label='Perceptron #0 decision boundary')
ax.plot(t1, decision_boundary(w1, t1), 'g', label='Perceptron #1 decision boundary')

where decision boundaries is

def decision_boundary(weights, x0):
    return -1. * weights[0]/weights[1] * x0
  • $\begingroup$ thanks - what if they were trained however? Would they converge to the same function or is it still possible to get the above result? $\endgroup$ – ashar Mar 3 '19 at 17:45
  • $\begingroup$ Training a neural network is minimizing a function (the loss). It is possible that if you trained your neural network with different initial weight they could not converge to the same weight and hence will not have the same minimum. $\endgroup$ – Robin Nicole Mar 3 '19 at 22:36

You have two forces working here: initial conditions and training.

Regarding training: in the end, we don't really care which of the two hidden-layer perceptrons become that near-vertical line in your second figure and which becomes the near-horizontal line - as long as convergence does happen.

Regarding which of the two perceptrons become which of the two decision boundaries in question, that takes us to network initialization.

See these lecture slides from the University of Sterling directly addressing your question. From slide 13:

The gradient descent learning algorithm treats all the weights in the same way, so if we start them all off with the same values, all the hidden units will end up doing the same thing and the network will never learn properly.

addressing your question. Followed by

For that reason, we generally start off all the weights with small random values. Usually we take them from a flat distribution around zero [–wt, +wt], or from a Gaussian distribution around zero with standard deviation wt.

Proposing a solution which I'm pretty sure is very common practice for these kind of networks. More useful notes on initialization are given the same slide.

Generally, there's a lot of effort invested in deciding what to do at initialization. Keras has many options around initializing, for example.

  • $\begingroup$ Side note: This is just one of a very long list of ways randomness in models helps learning. In NNs, dropout is kinda mind blowing. In random forests, it is often beneficial to only show any of the individual trees a subset of features and/or samples. $\endgroup$ – TheGrimmScientist Mar 3 '19 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.