# Why can it be that my neural network is predicting the contrary?

I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.

Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.

I'm training it with the following:

X = [[-3], [-2], [3], [2]]

Y = [[0], [0], [1.], [1.]]

but when I use that same X to check, it predicts aprox. [[1.], [1.], [0], [0]]

the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.

If I change Y to [1, 1, [0], [0]] the prediction is 0.5 for all instances, and both bias and weight is close to 0. With this Y configuration the cost stays at 0.69 through all iterations.

If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.

    def gradients(self, X, Y):
dws = []
dbs = []
h, activations, zs = self.forward(X)
da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput
for d in np.arange(1, len(self.weights))[::-1]:
a = activations[d] #output for layer d
z = zs[d] #linear for layer d
w = self.weights[d] #weights for layer w

dz = da * a * (1-a)
m = w.shape[1]
dw = 1./m * dz.dot(activations[d].T)
db = 1./m * np.sum(dz, axis=1, keepdims=True)
da = w.T.dot(dz)
dws.insert(0, dw)
dbs.insert(0, db)
dws.insert(0, None)
dbs.insert(0, None)
return dws, dbs


I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.

For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.

Here is the whole code: GitHub

dw = 1./m * dz.dot(activations[d].T)

dw = 1./m * dz.dot(activations[d-1].T)