2
$\begingroup$

Is there a way to reduce the noisiness and stochasticity of Rprop (and for that matter the iRprop+)?

Specifically, in deep networks (with 8+ layers) this effect starts to become apparent, as the earliest layers are adjusted. This has a massive effect on the outcome, and the error jumps around.

The noise also happens in a network with just 2-3 layers. However, it is only apparent when the error reached 0.00, and the iRprop+ keeps running. In some cases it will suddenly cause a very abrupt change, and the cross-entropy cost function will produce error larger than 1 000 000

I've built a custom LSTM in c++ and experiencing this noise during debugging of overfitting

Perhaps the learning rates from each layer should be initialized with a different magnitude, depending on the layer's depth? [link]

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ What is your batch size? Have you tried larger minibatch sizes? Related (but not an answer): datascience.stackexchange.com/questions/25024/… $\endgroup$
    – Neil Slater
    Commented Dec 22, 2017 at 7:50
  • $\begingroup$ I've just realized that Rprop is a batch learning algorithm - it requires predicting through entire dataset before performing error propagation. It actually worked because my alphabet prediction task happens only after all alphabet characters were predicted. So your comment made me think about using another error-prop algorithm $\endgroup$
    – Kari
    Commented Dec 22, 2017 at 9:25
  • $\begingroup$ Actually sorry I misread your question as being about RMSProp . . . rprop might suffer from similar issues. If you ran rprop on minibatches you could get some odd behaviour. $\endgroup$
    – Neil Slater
    Commented Dec 22, 2017 at 9:33

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.