2
$\begingroup$

Is there a way to reduce the noisiness and stochasticity of Rprop (and for that matter the iRprop+)?

Specifically, in deep networks (with 8+ layers) this effect starts to become apparent, as the earliest layers are adjusted. This has a massive effect on the outcome, and the error jumps around.

The noise also happens in a network with just 2-3 layers. However, it is only apparent when the error reached 0.00, and the iRprop+ keeps running. In some cases it will suddenly cause a very abrupt change, and the cross-entropy cost function will produce error larger than 1 000 000

I've built a custom LSTM in c++ and experiencing this noise during debugging of overfitting

Perhaps the learning rates from each layer should be initialized with a different magnitude, depending on the layer's depth? [link]

$\endgroup$
  • 1
    $\begingroup$ What is your batch size? Have you tried larger minibatch sizes? Related (but not an answer): datascience.stackexchange.com/questions/25024/… $\endgroup$ – Neil Slater Dec 22 '17 at 7:50
  • $\begingroup$ I've just realized that Rprop is a batch learning algorithm - it requires predicting through entire dataset before performing error propagation. It actually worked because my alphabet prediction task happens only after all alphabet characters were predicted. So your comment made me think about using another error-prop algorithm $\endgroup$ – Kari Dec 22 '17 at 9:25
  • $\begingroup$ Actually sorry I misread your question as being about RMSProp . . . rprop might suffer from similar issues. If you ran rprop on minibatches you could get some odd behaviour. $\endgroup$ – Neil Slater Dec 22 '17 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.