tf.nn.dynamic_rnn() and tf.nn.raw_rnn() take in an argument called parallel_iterations. The documentation says:

parallel_iterations: (Default: 32). The number of iterations to run in parallel. Those operations which do not have any temporal dependency and can be run in parallel, will be. This parameter trades off time for space. Values >> 1 use more memory but take less time, while smaller values use less memory but computations take longer.

If memory is not the constraint, should i not always keep its value equal to the number of processors available with me on GPU.

Here i have my first question - Am i right in thinking that number of processors on GPU mean number of parallel floating point computations it can undertake (i don't suppose GPU's too implement 2 threads per processors as in CPUs)?

Now, GRU machinery is: (please note the biases are not showcased in the image but are to be used for better performance)


To me the value 32 seems too low to be kept as default. If i have a GRU with 250 dimensional hidden state and 100 dimensional input i would have (2 x 250 x 350 x batch_size) floating multiplications that i can run in parallel (computing r and u) followed by (250 x batch_size) floating point additions that i can run in parallel followed by (250 x batch_size) sigmoid applications which are at least (250 x batch_size) potential parallel computations. Then it is a (250 x batch_size) parallel computations in application of reset gate followed by (1 x 250 x 350 x batch_size) parallel computations followed by (250 x batch_size) parallel computations for biases. Now in the final step it is (250 x batch_size) floating point computations to calculate (1 - update gate) followed by (500 x batch_size) parallel floating point computations for output state. To me it seems the bottlenecks in parallel implementations using GPU would be presence of too many low sized layers which, for instance here, are bias additions after matrix multiplications (at any rate they are still 250 x batch sized big here so will anyway use GPU to the maximum)

Is the way i am thinking in the second paragraph correct while analyzing a deep network for a vague idea on how much the hardware would be able to impact its running time

  • $\begingroup$ Did you consider to test and benchmark this? $\endgroup$
    – Frankstr
    Apr 2, 2018 at 14:06
  • $\begingroup$ I am implementing a Dynamic Memory Network as described arxiv.org/pdf/1603.01417.pdf and for some reason it is almost same speed on my cpu and k-80 tesla gpu. My cpu completes 60 iterations in the time gpu completes 80 and both slow down equally as i increase the batch size. Haven't been able to do much more as each epoch is taking 7 hours. I dont understand by they would take same time on cpu and gpu $\endgroup$ Apr 2, 2018 at 17:04
  • $\begingroup$ changing parallel iterations also changes nothing in relative speeds $\endgroup$ Apr 2, 2018 at 17:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.