# RMSProp Optimizer Performing Poorly

I am building an RNN and have decided to try RMSProp as an alternative to sgd. Here is my implementation:

self.R = {1:np.ones(self.W_xh[0].shape),
2:np.ones(self.W_hh[0].shape),
3:np.ones(self.W_hy[0].shape),
4:np.ones(self.b_h[0].shape),
5:np.ones(self.b_y[0].shape)}

self.lr = 1e-2
self.iota = 1e-6
self.gamma = 0.9
self.epochs = 10000

for e in range(self.epochs):
for p in range(batches):
if self.opt_alg == 'rmsprop':
i = 0
for param, dparam in zip([self.W_xh, self.W_hh, self.W_hy, self.b_h, self.b_y], [dW_xh, dW_hh, dW_hy, db_h, db_y]):
i = i + 1
self.R[i] = self.R[i] + np.multiply((1 - self.gamma), self.R[i]) + np.multiply(self.gamma, np.square(dparam))
param = param - np.multiply(self.lr, (dparam/np.sqrt(self.R[i] + self.iota)))


i increases so that each entry in the dict can be accessed with its respective parameter. 'param' is a reference to the weights and 'dparam' is a reference to the new gradient, to be applied to 'param' using RMSProp.

Here are some results from training:

Epoch 12, Iter 0: Loss: 88.8836
Epoch 13, Iter 0: Loss: 88.8836
Epoch 14, Iter 0: Loss: 88.8836
Epoch 15, Iter 0: Loss: 88.8836
Epoch 16, Iter 0: Loss: 88.8836


As you can see it seems to be stuck, and is actually underperforming compared to sgd. My data isn't too strenuous either - just around 40 words. How can I fix this, and have I implemented it correctly?