The backpropagation procedure is taken from the approach outlined in here. Here is the code, commented:

def sigma(z):
    return 1/(1+np.exp(-z))
def sigmaprime(z):
    return sigma(z) * (1- sigma(z))

class Network():
    def __init__(self, sizes) -> None:
        self.sizes= sizes 
        self.l = len(sizes) # length of NN, including input and output layers
        self.w = [np.random.randn(x, y) for x, y in zip(sizes[1:], sizes[:-1])] # initializes the weights
        self.b = [np.random.randn(y) for y in sizes[1:]] # initializes the biases

    def feedforward(self, a): # a simple feedforward
        for i in range(0, self.l-1):
            z = np.matmul(self.w[i], a) + self.b[i]
            a = sigma(z)
        return a
    def backprop(self, input, target):
        zlist =[]
        for i in range(0, self.l-1): # a simple feed forward, keeping track of the weighted inputs z and activated outputs a
            z = np.matmul(self.w[i], a) + self.b[i]
            a = sigma(z)
        print(np.linalg.norm(a-target)/2) # this is the mean square loss
        d = (a-target) * sigmaprime(zlist[-1]) # calculates the last 'delta' 
        for i in range(1, self.l-1): # backpropagates the delta
            d = np.matmul(self.w[-i].T, d) * sigmaprime(zlist[-i-1])
        gradb = dlist[::-1] # gradient for biases
        gradw = [ np.outer( dlist[-i], alist[i-1]) for i in range(1, self.l)] # gradient for weight matrices
        return gradw, gradb
    def train(self, input, target, lr): #trains via gradient descent on 1 input
        gradw, gradb= self.backprop(input, target)
        for i in range(0,self.l-1):
            self.w[i] -= lr * gradw[i]
            self.b[i] -= lr * gradb[i]

net = Network([3,4,1]) # 3 input neurons, 4 hidden neurons, 1 dimensional output

Here is the problem i'm trying to learn:

X = np.array([[0,0,0],
y = np.array([1,1,1,0,1,0,1,1]) # The problem I want to learn. An arbitrary function of 3 dimensional binary input and 1-dim binary output

epochs =1000 # training
from random import randint
for i in range(1,epochs):
    s = randint(0,len(X)-1)
    net.train(np.array(X[s]), np.array(y[s]), lr=.2)

After training the ouput is not right at all, but instead it seems to output the same value for every input. What is happening? The code seems right, it mimics the linked page's procedure. I've checked it multiple times and rewritten it.


1 Answer 1


There are a couple of issues, but I think the main one is your training loop. An epoch is generally one pass through the entire dataset, and then backprop is performed, notwithstanding mini-batch or SGD which we'll skip for now. You're choosing one input randomly, and running backprop on it, which means that the gradients are likely jumping around all over. A slight modification helps:

ind = [x for x in range(0,len(X))]

and then in the training loop,

for i in range(1,epochs):

    for s in ind:
    #s = randint(0,len(X)-1)
        net.train(np.array(X[s]), np.array(y[s]), lr=.1)

This mostly works for me:

  • input: [0 0 0], output: [0.98439051]
  • input: [0 0 1], output: [0.85588698]
  • input: [0 1 0], output: [0.98184023]
  • input: [0 1 1], output: [0.13676313]
  • input: [1 0 0], output: [0.9800416]
  • input: [1 0 1], output: [0.13235091]
  • input: [1 1 0], output: [0.95830572]
  • input: [1 1 1], output: [0.87568369]

A couple of other things: the learning rate is 0.2, which is still typically very high but works ok for this. Also input is a python defined function so I would recommend renaming the parameters to your functions. Last but not least, I'd recommend initializing random seeds for python and numpy:



  • $\begingroup$ Thank you very much for your time. Indeed it was the training loop. $\endgroup$ Sep 20 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.