1
$\begingroup$

I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation:

self.R = {1:np.ones(self.W_xh[0].shape),
         2:np.ones(self.W_hh[0].shape),
         3:np.ones(self.W_hy[0].shape),
         4:np.ones(self.b_h[0].shape),
         5:np.ones(self.b_y[0].shape)}

self.lr = 1e-2
self.iota = 1e-6
self.gamma = 0.9
self.epochs = 10000

for e in range(self.epochs):
    for p in range(batches):
        if self.opt_alg == 'rmsprop':
            i = 0
            for param, dparam in zip([self.W_xh, self.W_hh, self.W_hy, self.b_h, self.b_y], [dW_xh, dW_hh, dW_hy, db_h, db_y]):
                i = i + 1
                self.R[i] = self.R[i] + np.multiply((1 - self.gamma), self.R[i]) + np.multiply(self.gamma, np.square(dparam))
                param = param - np.multiply(self.lr, (dparam/np.sqrt(self.R[i] + self.iota)))

i increases so that each entry in the dict can be accessed with its respective parameter. 'param' is a reference to the weights and 'dparam' is a reference to the new gradient, to be applied to 'param' using RMSProp.

Here are some results from training:

Epoch 12, Iter 0: Loss: 88.8836
Epoch 13, Iter 0: Loss: 88.8836
Epoch 14, Iter 0: Loss: 88.8836
Epoch 15, Iter 0: Loss: 88.8836
Epoch 16, Iter 0: Loss: 88.8836

As you can see it seems to be stuck, and is actually underperforming compared to sgd. My data isn't too strenuous either - just around 40 words. How can I fix this, and have I implemented it correctly?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.