2
$\begingroup$

i have written this neural network for XOR function.the output is not correct.it is not classifying the test inputs correctly.can anyone please let me the reason why.

import numpy as np
import pandas as pd
x=np.array([[0,0],[0,1],[1,0],[1,1]])
y=np.array([[0],[1],[1],[0]])
np.random.seed(0) 
theta1=np.random.rand(2,8)
theta2=np.random.rand(8,1)
np.random.seed(0)
b1=np.random.rand(4,8)
b2=np.random.rand(4,1)
alpha=0.01
lamda=0.01

for i in range(1,2000):
    z1=x.dot(theta1)+b1

    h1=1/(1+np.exp(-z1))
    z2=h1.dot(theta2)+b2
    h2=1/(1+np.exp(-z2))

    dh2=h2-y
    #back prop

    dz2=dh2*(1-dh2)
    H1=np.transpose(h1)
    dw2=np.dot(H1,dz2)
    db2=np.sum(dz2)

    W2=np.transpose(theta2)
    dh1=np.dot(dz2,W2)
    dz1=dh1*(1-dh1)

    X=np.transpose(x)
    dw1=np.dot(X,dz1)

    db1=np.sum(dz1)


    dw2=dw2-lamda*theta2

    dw1=dw1-lamda*theta1
    theta1=theta1-alpha*dw1
    theta2=theta2-alpha*dw2
    b1+=-alpha*db1

    b2+=-alpha*db2


#prediction
#test inputs
input1=np.array([[0,0],[1,1],[0,1],[1,0]])
z1=np.dot(input1,theta1)

h1=1/(1+np.exp(-z1))
z2=np.dot(h1,theta2)

h2=1/(1+np.exp(-z2))

expected output=[0],[0],[1],[1] actual output=[[ 0.95678049] [ 0.99437206] [ 0.98686979] [ 0.98628204]]

all are ones here.

$\endgroup$

1 Answer 1

0
$\begingroup$

There are a few mistakes in the code, so I am going to present a revised version here with comments.

Setup

import numpy as np
import pandas as pd
x=np.array([[0,0],[0,1],[1,0],[1,1]])
y=np.array([[0],[1],[1],[0]])
np.random.seed(0)

# Optional, but a good idea to have +ve and -ve weights
theta1=np.random.rand(2,8)-0.5
theta2=np.random.rand(8,1)-0.5

# Necessary - the bias terms should have same number of dimensions
# as the layer. For some reason you had one bias vector per example.
# (You could still use np.random.rand(8) and np.random.rand(1))
b1=np.zeros(8)
b2=np.zeros(1)

alpha=0.01
# Regularisation not necessary for XOR, because you have a complete training set.
# You could have lamda=0.0, but I have left a value here just to show it works.
lamda=0.001

Training - Forward propagation

# More iterations than you might think! This is because we have
# so little training data, we need to repeat it a lot.
for i in range(1,40000):
    z1=x.dot(theta1)+b1
    h1=1/(1+np.exp(-z1))
    z2=h1.dot(theta2)+b2
    h2=1/(1+np.exp(-z2))

Training - Back propagation

    # This dz term assumes binary cross-entropy loss
    dz2 = h2-y 
    # You could also have stuck with squared error loss, the extra h2 terms
    # are the derivative of the sigmoid transfer function. 
    # It converges slower though:
    # dz2 = (h2-y) * h2 * (1-h2)

    # This is just the same as you had before, but with less temp variables
    dw2 = np.dot(h1.T, dz2)
    db2 = np.sum(dz2, axis=0)

    # The derivative of sigmoid is h1 * (1-h1), NOT dh1*(1-dh1)
    dz1 = np.dot(dz2, theta2.T) * h1 * (1-h1)
    dw1 = np.dot(x.T, dz1)
    db1 = np.sum(dz1, axis=0)

    # The L2 regularisation terms ADD to the gradients of the weights
    dw2 += lamda * theta2
    dw1 += lamda * theta1

    theta1 += -alpha * dw1
    theta2 += -alpha * dw2

    b1 += -alpha * db1
    b2 += -alpha * db2

Prediction

This is where you can kick yourself, you forgot to use the biases!

input1=np.array([[0,0],[1,1],[0,1],[1,0]])
z1=np.dot(input1,theta1)+b1
h1=1/(1+np.exp(-z1))
z2=np.dot(h1,theta2)+b2
h2=1/(1+np.exp(-z2))

print(h2)

When I run the above code I get a correct-looking output

[[ 0.01031446]
 [ 0.0201576 ]
 [ 0.9824826 ]
 [ 0.98584079]]

In summary your three big errors were the wrong dimension for the bias vectors in setup, incorrect derivatives for sigmoid function (using correct form, but with wrong variable) and forgetting to use bias at all when predicting at the end. Other details are still worth noting, but would not have prevented you getting something working.

$\endgroup$
1
  • $\begingroup$ this was so helpful.can't thank you in words @Neil $\endgroup$
    – user34042
    Jul 5, 2017 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.