I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.
Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.
I'm training it with the following:
X = [[-3], [-2], , ]
Y = [, , [1.], [1.]]
but when I use that same X to check, it predicts aprox. [[1.], [1.], , ]
the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.
If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.
This is my gradrient function
def gradients(self, X, Y): dws =  dbs =  h, activations, zs = self.forward(X) da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput for d in np.arange(1, len(self.weights))[::-1]: a = activations[d] #output for layer d z = zs[d] #linear for layer d w = self.weights[d] #weights for layer w dz = da * a * (1-a) m = w.shape dw = 1./m * dz.dot(activations[d].T) db = 1./m * np.sum(dz, axis=1, keepdims=True) da = w.T.dot(dz) dws.insert(0, dw) dbs.insert(0, db) dws.insert(0, None) dbs.insert(0, None) return dws, dbs
I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.
For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.
Here is the whole code: GitHub