Purpose of weights in neural networks

Question

I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing.

I know, that before the inputs are summed and passed to activation functions, they are separately weighted, after some research, i found out that the purpose of the weight function was to:

• ensure orthonormality
• avoid data loss

I know that two inputs are only orthonormal if they are orthogonal unit vectors, if they are orthogonal then their dot product is always 0. Dot product:

where theta is angle between two unit vectors and absolute value of a is norm.

For example, if two unit vectors are perpendicular to each other, then they are orthonormal and their dot product is 0.

But what does this have to do with data loss? I've also heard that sometimes the value of input might be zero, and we know that multiplication by zero outputs zero.

So what is the real purpose of the weight in neural networks? and what does it have to do with orthonormality? For example, what would be the purpose of weights in linear regression?

Answer 1

The reason for weights in machine learning is actually a lot easier than it seems. It's the way by which our model learns some underlining function and performs the classification or regression. We tune these weights in order to model some underlining function which can map our input to a desired output. Either a class in classification, or a range of values in a regression.

Let's look at an easier machine learning model so that we can understand why weights are needed.

Linear Regression

This is just a straight line which splits data into two sections. Let us apply this model in 2D, where we have two features $x = [x_1, x_2]$, for example weight and height. And labels $y$ which will split our data into $y \in \{men, women\}$.

A random line in this space is defined as

$0 = -2x_2 + x_1 + 1$.

Assume this is our boundary line. Let's just assume that women fall above this line and men fall below it. Like this

Now if we get a new data point $x_{new} = [2, 8]$ we will label this as being a woman. So the entire decision is based on the numbers $1, -2, 1$ from our linear equation. We need to tune these values using the training data. We usually call these trainable parameters the weights $w$ associated with the features $x$ and we also add a bias $b$. In general the linear separator in 2D is

$0 = w_1x_1 + w_2x_2 + b$.

Our predicted label is $\hat{y} = w_1x_1 + w_2x_2 + b$. If $\hat{y} > 0$ then woman, else man.

In $n$ dimensions this can be written as a matrix multiplication as

$\hat{y} = w^Tx + b$

Obviously, a linear separator is not sufficient for most classification tasks. Things are not always linearly separable. So we use more complex models.

Neural networks

In neural networks each node is associated with a function much like the linear separator. However, we will use the sigmoid function

$\sigma(w^Tx) = \frac{1}{1 + exp^{-(w^Tx + b)}}$.

The weights here have the same effect. They will modulate the input values $x$ such that we are able to learn some classification or regression.

---

How do the weights affect the decision boundary. We want the two different classes, circles and x's, to be on opposite sides of this boundary in order for us to be able to correctly classify them.

The weights are trained iteratively using gradient descent. We can see that the decision boundary starts off terribly and then gets progressively better.

$0 = -1.0 - 1.0x_1 - 1.0x_2$

$0 = -16.0 - -39.0x_1 - 71.0x_2$

$0 = -36.0 - -94.0x_1 - 61.0x_2$

$0 = -83.5 - -134.0x_1 - 76.0x_2$

$0 = -88.5 - -114.0x_1 - 98.5x_2$

As you can see we changed the weights until we were able to find this ideal boundary between the two classes. If you want to find our more about how the gradient descent algorithm works to tune these weights you can look here.