# Why my perceptron doesn't train well and produces bad results on test data?

I am a newbie in Machine learning and I am writing a small code for Perceptron. This is the first time I am writing code in Python. I have four training data points (X). As they are used for supervised learning so, each data point has its corresponding correct output pair (D). I have implemented SGD and used generalized Delta rule (wij ← wij + α δixj). I have trained my perceptron 10,000 times (epochs= 10,000).
Although everything looks fine to me, I don't get the right results when I test it with test values. I need some suggestions so that I can improve my results on test data. P.S. How can I improve this code?

Code

import numpy as np

def sigmoid(x):
return 1 / (1 + np.exp(-x))

def Delta_SGD(W, X, D):
N = 4
for x in range(N):

v1 = np.dot(X[x][0], W[0])
v2 = np.dot(X[x][1], W[1])
v3 = np.dot(X[x][2], W[2])
#weighted sum
V = v1+v2+v3

#output of neuron
y = sigmoid(V)

#error
e = D[x] - y

#derivative of sigmoid(y)
delta = (y)*(1-y)*e

#Delta rule
DW = alpha*delta*X[x]

#updated weights
W[0] = W[0] + DW[0]
W[1] = W[1] + DW[1]
W[2] = W[2] + DW[2]

return W

#input data points
X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])

#Correct output pairs
D = np.array([[0,0,1,1]]).T

#learning rate
alpha = 0.9

#random weights
W =  2*np.random.random((3,1)) - 1

#10000 epochs
for epoch in range(10000):
W = Delta_SGD(W, X, D)
print(epoch)

#Final weights after all epochs
print("Final weights are \n", W)

#testing network
N = 4
for x in range(N):

v1 = np.dot(X[x][0], W[0])
v2 = np.dot(X[x][1], W[1])
v3 = np.dot(X[x][2], W[2])

V = v1+v2+v3
y = sigmoid(V)
print("output of neuron is \n ", y)

• your training data consists of only 4 samples? – oW_ Oct 18 '18 at 16:46
• yes only four samples as its a basic perceptron. – Ammar Ul Hassan Oct 19 '18 at 0:00
• without looking at your code you don't have enough samples, in particular for 10,000 steps. there is only so much to learn from four samples... you'd have a better chance with 10,000 samples and 4 epochs – oW_ Oct 19 '18 at 15:15
• Everything is right my friend. show us your test data. maybe that is the problem. I tried your code and it works. – Kasra Manshaei Oct 21 '18 at 14:02
• @KasraManshaei Thanks it's working perfectly now. The test data I used is the same as the input as u can see in the above code. – Ammar Ul Hassan Oct 22 '18 at 7:28

Two points about the whole thing

1. You did not test yet. The point behind the training process is to make machine able to learn from the data conditioned on the ability of generalizing this to predicting samples which it has not seen before. Otherwise, the good training is actually overfitting. So here you trained on X and you need to create new samples and check the result to really call it testing (this is an introductory explanation).
2. Machine Learning is about features a lot! Playing with features and cleaning, modifying and filtering them is a key point. In your example, the last dimension of your 3d data is always 1. Does it distinguish anything? (in my course you get a complete explanation of this in the Lecture 2). So that feature (dimenstion, element of the vector) can/should be removed. To better understanding, imagine your 3d spread of the data. The z axis is always 1 which means the topology of points is what you see in x-y plain. So use only that one.
• Yeah, you are absolutely right. I just did it for purpose and that is that I wanted to learn how 3 inputs can be used to input a single neuron. Otherwise, if I just remove the third column I can tweak these inputs for the OR and AND problems etc. And for the first point, I totally agree but I am learning step by step :) BTW i'll look into your course. – Ammar Ul Hassan Oct 23 '18 at 13:57
• As a starter your understanding and code are both very elegant! Good Luck my friend :) – Kasra Manshaei Oct 23 '18 at 13:59