Bugs in the backpropagation algorithm in Python

Question

I've been trying to create a simple Neural Network from scratch with a backpropagation algorithm to predict the next number based on 3 previous numbers. But for some reasons, MSE(Mean Squared Error) becomes +- the same in each epoch after some point, while the difference between a predicted number and an actual number is still quite large. Can anyone do a code review for possible bugs and explain why the Neural Network is not learning? Thanks.

import numpy as np


def prediction():
    def ReLU(x):
        return np.maximum(0, x)

    ReLU = np.vectorize(ReLU)

    def MSE(Y, y):
        return (Y - y) ** 2

    MSE = np.vectorize(MSE)

    x = np.array(
        [
            [2.16, 3.19, 1.85],
            [3.19, 1.85, 4.84],
            [1.85, 4.84, 0.55],
            [4.84, 0.55, 4.20],
            [0.55, 4.20, 1.68],
            [4.20, 1.68, 4.74],
            [1.68, 4.74, 0.14],
            [4.74, 0.14, 5.68],
            [0.14, 5.68, 0.48],
            [5.68, 0.48, 5.03],
        ]
    )

    y = np.array([4.84, 0.55, 4.20, 1.68, 4.74, 0.14, 5.68, 0.48, 5.03, 0.18])

    xt = np.array(
        [
            [0.48, 5.03, 0.18],
            [5.03, 0.18, 5.99],
        ]
    )

    yt = np.array([5.99, 0.09])

    w1 = np.random.uniform(0, 1, (4, 3))
    w2 = np.random.uniform(0, 1, (4,))

    global E
    E = 0
    v = 0.005
    epoch = 0

    # Training
    while epoch < 10_000:
        s1 = np.dot(w1, x.T)
        y1 = ReLU(s1)

        s2 = np.dot(w2, y1)
        y2 = ReLU(s2)

        Ei = np.sum(MSE(y2, y)) / (2 * len(y2))
        print(Ei)
        if Ei < 0.05:
            print(w1)
            print(w2)
            break

        delta_w2 = np.dot(y2 - y, y1.T)
        delta_w1 = np.dot(np.dot(w2.reshape(4, 1), (y2 - y).reshape(1, 10)), x)

        w2 = w2 - v * delta_w2
        w1 = w1 - v * delta_w1

        E = Ei
        epoch += 1

    # Testing
    s1 = np.dot(w1, xt.T)
    y1 = ReLU(s1)

    s2 = np.dot(w2, y1)
    y2 = ReLU(s2)

    print(f"- A {yt}")
    return y2


print(f"- P {prediction()}")

Some of the possible results, where A is an actual number, P is a predicted number:

1. Why is the second number 0?

• A [5.99 0.09]
• P [5.13884088 0. ]

2. This one looks like a dying ReLU problem because MSE becomes a static 6.22519 number for each epoch, but I'm not sure. How do I fix this?

• A [5.99 0.09]
• P [0. 0.]

3. One of the closest results, although the difference is still quite large.

• A [5.99 0.09]
• P [5.58113987 0.33711845]

Also, a few resources from which I studied the algorithm:
http://neuralnetworksanddeeplearning.com/chap2.html
https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
http://galaxy.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html