This simple python Feed forward Neural Network isn't learning. What am I doing wrong?

Question

The backpropagation procedure is taken from the approach outlined in here. Here is the code, commented:

def sigma(z):
    return 1/(1+np.exp(-z))
def sigmaprime(z):
    return sigma(z) * (1- sigma(z))

class Network():
    def __init__(self, sizes) -> None:
        
        self.sizes= sizes 
        self.l = len(sizes) # length of NN, including input and output layers
        self.w = [np.random.randn(x, y) for x, y in zip(sizes[1:], sizes[:-1])] # initializes the weights
        self.b = [np.random.randn(y) for y in sizes[1:]] # initializes the biases


    def feedforward(self, a): # a simple feedforward
        for i in range(0, self.l-1):
            z = np.matmul(self.w[i], a) + self.b[i]
            a = sigma(z)
        return a
    
    def backprop(self, input, target):
        a=input
        zlist =[]
        alist=[a]
        for i in range(0, self.l-1): # a simple feed forward, keeping track of the weighted inputs z and activated outputs a
            z = np.matmul(self.w[i], a) + self.b[i]
            zlist.append(z)
            a = sigma(z)
            alist.append(a)
        
        print(np.linalg.norm(a-target)/2) # this is the mean square loss
        d = (a-target) * sigmaprime(zlist[-1]) # calculates the last 'delta' 
        dlist=[d]
        
        for i in range(1, self.l-1): # backpropagates the delta
            d = np.matmul(self.w[-i].T, d) * sigmaprime(zlist[-i-1])
            dlist.append(d)
        #print(dlist)
        gradb = dlist[::-1] # gradient for biases
        gradw = [ np.outer( dlist[-i], alist[i-1]) for i in range(1, self.l)] # gradient for weight matrices
        return gradw, gradb
    
    def train(self, input, target, lr): #trains via gradient descent on 1 input
        gradw, gradb= self.backprop(input, target)
        for i in range(0,self.l-1):
            self.w[i] -= lr * gradw[i]
            self.b[i] -= lr * gradb[i]



net = Network([3,4,1]) # 3 input neurons, 4 hidden neurons, 1 dimensional output
        

Here is the problem i'm trying to learn:

X = np.array([[0,0,0],
              [0,0,1],
              [0,1,0],
              [0,1,1],
              [1,0,0],
              [1,0,1],
              [1,1,0],
              [1,1,1]])
    
y = np.array([1,1,1,0,1,0,1,1]) # The problem I want to learn. An arbitrary function of 3 dimensional binary input and 1-dim binary output

epochs =1000 # training
from random import randint
for i in range(1,epochs):
    s = randint(0,len(X)-1)
    net.train(np.array(X[s]), np.array(y[s]), lr=.2)
    

After training the ouput is not right at all, but instead it seems to output the same value for every input. What is happening? The code seems right, it mimics the linked page's procedure. I've checked it multiple times and rewritten it.

Answer 1

There are a couple of issues, but I think the main one is your training loop. An epoch is generally one pass through the entire dataset, and then backprop is performed, notwithstanding mini-batch or SGD which we'll skip for now. You're choosing one input randomly, and running backprop on it, which means that the gradients are likely jumping around all over. A slight modification helps:

ind = [x for x in range(0,len(X))]

and then in the training loop,

for i in range(1,epochs):
    random.shuffle(ind)

    for s in ind:
    #s = randint(0,len(X)-1)
        net.train(np.array(X[s]), np.array(y[s]), lr=.1)

This mostly works for me:

• input: [0 0 0], output: [0.98439051]
• input: [0 0 1], output: [0.85588698]
• input: [0 1 0], output: [0.98184023]
• input: [0 1 1], output: [0.13676313]
• input: [1 0 0], output: [0.9800416]
• input: [1 0 1], output: [0.13235091]
• input: [1 1 0], output: [0.95830572]
• input: [1 1 1], output: [0.87568369]

A couple of other things: the learning rate is 0.2, which is still typically very high but works ok for this. Also input is a python defined function so I would recommend renaming the parameters to your functions. Last but not least, I'd recommend initializing random seeds for python and numpy:

np.random.seed(42)
random.seed(42)

hth.