In particular, how much memory does a recurrent NN require as a function of the dataset size,number of nodes, etc., and how expensive is it to evaluate at runtime given a new test point?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I've found some time ago two interesting papers about recurrent neural networks and their complexity. I guess you can use those as a reference points at least:
$\begingroup$
$\endgroup$
3
Besides the excellent references given by sebap123, from the Deep Learning Book by Ian Goodfellow et.al,
The recurrent neural network [given] is universal in the sense that any function computable by a Turing machine can be computed by such a recurrent network of a finite size. The output can be read from the RNN after a number of time steps that is asymptotically linear in the number of time steps used by the Turing machine and asymptotically linear in the length of the input .
Hope this helps!
-
$\begingroup$ I had to re-read that quote to figure out the link to the question . . . theoretical comparisons to a Turing machine are some steps away from specifics in the question, such as number of nodes vs memory use. $\endgroup$ Apr 27, 2017 at 12:57
-
$\begingroup$ My bad, I should have read the question details. I was thinking about asymptotic complexity. $\endgroup$– ngub05Apr 27, 2017 at 13:08
-
$\begingroup$ Well I guess if someone searches and finds this on the title alone, then they will find your answer which could be useful to them, so perhaps leave it as is. $\endgroup$ Apr 27, 2017 at 13:10