I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation:

self.R = {1:np.ones(self.W_xh[0].shape),

self.lr = 1e-2
self.iota = 1e-6
self.gamma = 0.9
self.epochs = 10000

for e in range(self.epochs):
    for p in range(batches):
        if self.opt_alg == 'rmsprop':
            i = 0
            for param, dparam in zip([self.W_xh, self.W_hh, self.W_hy, self.b_h, self.b_y], [dW_xh, dW_hh, dW_hy, db_h, db_y]):
                i = i + 1
                self.R[i] = self.R[i] + np.multiply((1 - self.gamma), self.R[i]) + np.multiply(self.gamma, np.square(dparam))
                param = param - np.multiply(self.lr, (dparam/np.sqrt(self.R[i] + self.iota)))

i increases so that each entry in the dict can be accessed with its respective parameter. 'param' is a reference to the weights and 'dparam' is a reference to the new gradient, to be applied to 'param' using RMSProp.

Here are some results from training:

Epoch 12, Iter 0: Loss: 88.8836
Epoch 13, Iter 0: Loss: 88.8836
Epoch 14, Iter 0: Loss: 88.8836
Epoch 15, Iter 0: Loss: 88.8836
Epoch 16, Iter 0: Loss: 88.8836

As you can see it seems to be stuck, and is actually underperforming compared to sgd. My data isn't too strenuous either - just around 40 words. How can I fix this, and have I implemented it correctly?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.