How to get the weights of a linear model by solving normal equation?

Question

In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168).

Linear Model:

$f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$

MSE Loss:

$J(\mathbf{w},b)= \frac{1}{4} \sum(f^*(\mathbf{x})-f(\mathbf{x;\mathbf{w},b})) $ , where $f^*(\mathbf{x})$ is the XOR function.

Normal equation:

According to the same book on page 107, the weights can be obtained by solving the gradient of the loss function, which will result in a normal equation (5.12).

$\mathbf{w}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$

My Attempts:

Since it is a XOR function, we know that if the input is $\mathbf{X}=\begin{bmatrix} 0 & 0 \\ 0 & 1 \\ 1 & 0 \\ 1 & 1 \\ \end{bmatrix}$, then the corresponding output will be $\mathbf{y}=\begin{bmatrix} 0 \\ 1 \\ 1 \\ 0 \\ \end{bmatrix}$. So I just plug everything into the normal equation as shown above.

However, the solution I get is $\mathbf{w}= \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \\ \end{bmatrix}$.

What am I doing wrong here? And also how to find the bias $b$?

Answer 1

Hey I know this is a year old but I wanted to provide the answer in case you haven't got it. So page 124 says how we derive the normal equations, however the following few paragraphs explains how to deal with a bias term.

You've got our dataset right, however we need to add an extra 1 to take the place of our bias in the weight vector. I'll append the ones at the end.

$\mathbf{X} = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 1 & 1 \\ \end{bmatrix} $

You've got our $y$ vector correct though, i'll write it down below for completeness sakes

$\mathbf{y} = \begin{bmatrix} 0 \\ 1 \\ 1 \\ 0 \\ \end{bmatrix} $

Now we compute solution to the normal equations, $w = (X^{T}X)^{-1}X^{t}y$, below

$\mathbf{w} = \begin{bmatrix} 0 \\ 0 \\ \frac{1}{2} \\ \end{bmatrix}$

Now the element in the 3rd position of our vector is the bias term and hence $b = \frac{1}{2}$ and the first two element of $\mathbf{w}$ is our weight vector equalling zero.

Hopefully you find this useful!