# Bugs in the backpropagation algorithm in Python

I've been trying to create a simple Neural Network from scratch with a backpropagation algorithm to predict the next number based on 3 previous numbers. But for some reasons, MSE(Mean Squared Error) becomes +- the same in each epoch after some point, while the difference between a predicted number and an actual number is still quite large. Can anyone do a code review for possible bugs and explain why the Neural Network is not learning? Thanks.

import numpy as np

def prediction():
def ReLU(x):
return np.maximum(0, x)

ReLU = np.vectorize(ReLU)

def MSE(Y, y):
return (Y - y) ** 2

MSE = np.vectorize(MSE)

x = np.array(
[
[2.16, 3.19, 1.85],
[3.19, 1.85, 4.84],
[1.85, 4.84, 0.55],
[4.84, 0.55, 4.20],
[0.55, 4.20, 1.68],
[4.20, 1.68, 4.74],
[1.68, 4.74, 0.14],
[4.74, 0.14, 5.68],
[0.14, 5.68, 0.48],
[5.68, 0.48, 5.03],
]
)

y = np.array([4.84, 0.55, 4.20, 1.68, 4.74, 0.14, 5.68, 0.48, 5.03, 0.18])

xt = np.array(
[
[0.48, 5.03, 0.18],
[5.03, 0.18, 5.99],
]
)

yt = np.array([5.99, 0.09])

w1 = np.random.uniform(0, 1, (4, 3))
w2 = np.random.uniform(0, 1, (4,))

global E
E = 0
v = 0.005
epoch = 0

# Training
while epoch < 10_000:
s1 = np.dot(w1, x.T)
y1 = ReLU(s1)

s2 = np.dot(w2, y1)
y2 = ReLU(s2)

Ei = np.sum(MSE(y2, y)) / (2 * len(y2))
print(Ei)
if Ei < 0.05:
print(w1)
print(w2)
break

delta_w2 = np.dot(y2 - y, y1.T)
delta_w1 = np.dot(np.dot(w2.reshape(4, 1), (y2 - y).reshape(1, 10)), x)

w2 = w2 - v * delta_w2
w1 = w1 - v * delta_w1

E = Ei
epoch += 1

# Testing
s1 = np.dot(w1, xt.T)
y1 = ReLU(s1)

s2 = np.dot(w2, y1)
y2 = ReLU(s2)

print(f"- A {yt}")
return y2

print(f"- P {prediction()}")


Some of the possible results, where A is an actual number, P is a predicted number:

1. Why is the second number 0?
• A [5.99 0.09]
• P [5.13884088 0. ]
1. This one looks like a dying ReLU problem because MSE becomes a static 6.22519 number for each epoch, but I'm not sure. How do I fix this?
• A [5.99 0.09]
• P [0. 0.]
1. One of the closest results, although the difference is still quite large.
• A [5.99 0.09]
• P [5.58113987 0.33711845]

Also, a few resources from which I studied the algorithm:
http://neuralnetworksanddeeplearning.com/chap2.html
https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
http://galaxy.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html

• I have some questions before looking at your code in detail: (1) What is the function / logic behind your data? Would it be an option to generate more training data? (2) Since you have few training points, no regulation and a lot of epochs, I do see the risk of overfitting. Did you have a look how the network performs on your training data? Mar 18 at 18:18