I have developed an RNN in matlab and now its time to test it. You can set your desired number of layers and nodes and it trains data in random chunks. I also included an annealing function for learning rate that decays gradually from 1 to 1e-5 to avoid overtraining of data.

I would like you to suggest me some simple ways to test my network in a way that i can get some useful feedback about its behavior from the results. I am a little concerned about convergence. I' ve set to do some testing of my own but would like to hear some suggestions from more experienced users.

I try to simulate some 1st, 2nd, 3rd, ... grade function between (0-1) and see how it goes. I will also try to predict the next value of a time series based on previous values, as another testing method. What would be a good result from these tests? An accuracy of ??% and an mse of ?? ?

  • $\begingroup$ Before you test with data: Have you tested the weight gradients by checking against measured gradients? Worth doing if you have implemented backprop algorithm yourself. $\endgroup$ – Neil Slater Jun 6 '16 at 18:25
  • $\begingroup$ I have implemented a BPTT algorithm. But can you be more precise about the procedure? Where can i find the measured gradients? $\endgroup$ – lopsi Jun 6 '16 at 21:24
  • $\begingroup$ You measure an approximate gradient for a parameter by calculating the loss for two slightly different values of the param (e.g. change a single weight by +1.0e-3) over the data set whilst keeping all other params the same. I.e. feed forward the network as normal on a small sample data set with two very slightly different weights. Then divide the difference between the loss by the difference you used for the weight param. This value should be close to the value you calculated using BPTT. Do for all weights in a simple network, one by one. This is a kind of unit test for your BPTT algorithm. $\endgroup$ – Neil Slater Jun 7 '16 at 7:42

When I built my first vanilla RNN from scratch, I was very curious about it's correctness. So, I trained it on a portion of a sine curve and tried to generate the part of the curve that comes next. And consequently, I plotted mine and the actual to visualise.

(P.S.: here is my code!)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.