I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing.

I know, that before the inputs are summed and passed to activation functions, they are separately weighted, after some research, i found out that the purpose of the weight function was to:

  • ensure orthonormality
  • avoid data loss

I know that two inputs are only orthonormal if they are orthogonal unit vectors, if they are orthogonal then their dot product is always 0. Dot product:

enter image description here

where theta is angle between two unit vectors and absolute value of a is norm.

For example, if two unit vectors are perpendicular to each other, then they are orthonormal and their dot product is 0.

But what does this have to do with data loss? I've also heard that sometimes the value of input might be zero, and we know that multiplication by zero outputs zero.

So what is the real purpose of the weight in neural networks? and what does it have to do with orthonormality? For example, what would be the purpose of weights in linear regression?


1 Answer 1


The reason for weights in machine learning is actually a lot easier than it seems. It's the way by which our model learns some underlining function and performs the classification or regression. We tune these weights in order to model some underlining function which can map our input to a desired output. Either a class in classification, or a range of values in a regression.

Let's look at an easier machine learning model so that we can understand why weights are needed.

Linear Regression

This is just a straight line which splits data into two sections. Let us apply this model in 2D, where we have two features $x = [x_1, x_2]$, for example weight and height. And labels $y$ which will split our data into $y \in \{men, women\}$.

A random line in this space is defined as

$0 = -2x_2 + x_1 + 1$.

Assume this is our boundary line. Let's just assume that women fall above this line and men fall below it. Like this

enter image description here

Now if we get a new data point $x_{new} = [2, 8]$ we will label this as being a woman. So the entire decision is based on the numbers $1, -2, 1$ from our linear equation. We need to tune these values using the training data. We usually call these trainable parameters the weights $w$ associated with the features $x$ and we also add a bias $b$. In general the linear separator in 2D is

$0 = w_1x_1 + w_2x_2 + b$.

Our predicted label is $\hat{y} = w_1x_1 + w_2x_2 + b$. If $\hat{y} > 0$ then woman, else man.

In $n$ dimensions this can be written as a matrix multiplication as

$\hat{y} = w^Tx + b$

Obviously, a linear separator is not sufficient for most classification tasks. Things are not always linearly separable. So we use more complex models.

Neural networks

In neural networks each node is associated with a function much like the linear separator. However, we will use the sigmoid function

$\sigma(w^Tx) = \frac{1}{1 + exp^{-(w^Tx + b)}}$.

The weights here have the same effect. They will modulate the input values $x$ such that we are able to learn some classification or regression.

How do the weights affect the decision boundary. We want the two different classes, circles and x's, to be on opposite sides of this boundary in order for us to be able to correctly classify them.

The weights are trained iteratively using gradient descent. We can see that the decision boundary starts off terribly and then gets progressively better.

$0 = -1.0 - 1.0x_1 - 1.0x_2$

enter image description here

$0 = -16.0 - -39.0x_1 - 71.0x_2$

enter image description here

$0 = -36.0 - -94.0x_1 - 61.0x_2$

enter image description here

$0 = -83.5 - -134.0x_1 - 76.0x_2$

enter image description here

$0 = -88.5 - -114.0x_1 - 98.5x_2$

enter image description here

As you can see we changed the weights until we were able to find this ideal boundary between the two classes. If you want to find our more about how the gradient descent algorithm works to tune these weights you can look here.

  • $\begingroup$ Thank you for your great answer! Just have a few questions. As being new to Data Science, I thought y-axis was used for the output of the function. Is $−2x_2+x_1+1$ linear separator? Also why do we use these weights for inputs? $\endgroup$
    – ShellRox
    Mar 7, 2018 at 15:35
  • 1
    $\begingroup$ Yes, $y$ is the ground truth output. The one in your $y$ vector in your training data. $\hat{y}$ is predicted class that your model gives. Also, yes $−2x_2+x_1+1 = 0$ is a linear separator. You can try to plot this with MATLAB or python and you'll see. The inputs are $x$, the weights $w$ are a part of the model. $\endgroup$
    – JahKnows
    Mar 7, 2018 at 15:41
  • $\begingroup$ Apologies for being confused, what do weights and bias do here? Do they increase/decrease importance of these input values? Also how are they chosen for each input? Are they chosen according to the dot products of all inputs (to ensure orthonormality)? $\endgroup$
    – ShellRox
    Mar 7, 2018 at 15:47
  • 1
    $\begingroup$ I'll add to the answer so you can see what is happening. $\endgroup$
    – JahKnows
    Mar 7, 2018 at 15:51
  • 1
    $\begingroup$ Weights are responsible for the predicted class. And the difference between the predicted class and the real one is going to affect your loss function. Actually, backpropagation uses gradient descent. Backpropagation is used when we have more than a single layer of neurons because the error is backpropagated across the layers from the output back. $\endgroup$
    – JahKnows
    Mar 7, 2018 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.