0
$\begingroup$

In chapter 6.1 of the book Deep Learning, the author tries to learn the XOR function by using a linear model (on page 168).

Linear Model:

$f(\mathbf{x};\mathbf{w},b)=\mathbf{x}^T\mathbf{w}+b$

MSE Loss:

$J(\mathbf{w},b)= \frac{1}{4} \sum(f^*(\mathbf{x})-f(\mathbf{x;\mathbf{w},b})) $ , where $f^*(\mathbf{x})$ is the XOR function.

Normal equation:

According to the same book on page 107, the weights can be obtained by solving the gradient of the loss function, which will result in a normal equation (5.12).

$\mathbf{w}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$

My Attempts:

Since it is a XOR function, we know that if the input is $\mathbf{X}=\begin{bmatrix} 0 & 0 \\ 0 & 1 \\ 1 & 0 \\ 1 & 1 \\ \end{bmatrix}$, then the corresponding output will be $\mathbf{y}=\begin{bmatrix} 0 \\ 1 \\ 1 \\ 0 \\ \end{bmatrix}$. So I just plug everything into the normal equation as shown above.

However, the solution I get is $\mathbf{w}= \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \\ \end{bmatrix}$.

What am I doing wrong here? And also how to find the bias $b$?

$\endgroup$
  • $\begingroup$ The X matrix should contain a first column which is equal to 1 in each row. This is the intercept of the model. $\endgroup$ – Peter Jun 20 '19 at 11:18
  • $\begingroup$ Here is a numerical example stats.stackexchange.com/a/80889/224077 $\endgroup$ – Peter Jun 20 '19 at 11:20
2
$\begingroup$

Hey I know this is a year old but I wanted to provide the answer in case you haven't got it. So page 124 says how we derive the normal equations, however the following few paragraphs explains how to deal with a bias term.

You've got our dataset right, however we need to add an extra 1 to take the place of our bias in the weight vector. I'll append the ones at the end.

$\mathbf{X} = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 0 & 1 \\ 1 & 1 & 1 \\ \end{bmatrix} $

You've got our $y$ vector correct though, i'll write it down below for completeness sakes

$\mathbf{y} = \begin{bmatrix} 0 \\ 1 \\ 1 \\ 0 \\ \end{bmatrix} $

Now we compute solution to the normal equations, $w = (X^{T}X)^{-1}X^{t}y$, below

$\mathbf{w} = \begin{bmatrix} 0 \\ 0 \\ \frac{1}{2} \\ \end{bmatrix}$

Now the element in the 3rd position of our vector is the bias term and hence $b = \frac{1}{2}$ and the first two element of $\mathbf{w}$ is our weight vector equalling zero.

Hopefully you find this useful!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.