0
$\begingroup$

I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.

Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.

I'm training it with the following:

X = [[-3], [-2], [3], [2]]

Y = [[0], [0], [1.], [1.]]

but when I use that same X to check, it predicts aprox. [[1.], [1.], [0], [0]]

the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.

If I change Y to [1, 1, [0], [0]] the prediction is 0.5 for all instances, and both bias and weight is close to 0. With this Y configuration the cost stays at 0.69 through all iterations.

If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.

This is my gradrient function

    def gradients(self, X, Y):
    dws = []
    dbs = []
    h, activations, zs = self.forward(X)
    da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput
    for d in np.arange(1, len(self.weights))[::-1]:
        a = activations[d] #output for layer d
        z = zs[d] #linear for layer d
        w = self.weights[d] #weights for layer w

        dz = da * a * (1-a)
        m = w.shape[1]
        dw = 1./m * dz.dot(activations[d].T)
        db = 1./m * np.sum(dz, axis=1, keepdims=True)
        da = w.T.dot(dz)
        dws.insert(0, dw)
        dbs.insert(0, db)
    dws.insert(0, None)
    dbs.insert(0, None)
    return dws, dbs

I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.

For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.

Here is the whole code: GitHub

$\endgroup$
1
$\begingroup$

After being all day trying to figure this out I had to post this question to find clarity. The problem is here

dw = 1./m * dz.dot(activations[d].T)

it should be

dw = 1./m * dz.dot(activations[d-1].T)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.