2
$\begingroup$

I'm running Keras' LSTM (not CuDNNLSTM) but I notice my GPU is under load. I need recurrent dropout, so I can only stick with LSTM. Is the 'normal' LSTM assisted by GPU? If so, how are LSTM and CuDNNLSTM different? I presume CuDNNLSTM uses the CUDNN API (and LSTM doesn't?

Similarly, is the normal LSTM supposed to be faster running on GPU or CPU?

$\endgroup$
1
$\begingroup$

Not sure if you figured this out, but I've been looking into it recently, and this is what I've found:

Is the 'normal' LSTM assisted by GPU?

is the normal LSTM supposed to be faster running on GPU or CPU?

Like @pcko1 said, LSTM is assisted by GPU if you have tensorflow-gpu installed, but it does not necessarily run faster on a GPU. In my case, it actually slowed it down by ~2x, because the LSTM is relatively small and the amount of copying between CPU and GPU made the training slower. I think with a larger network, it would speed things up. I also found that LSTM only used ~25% of the GPU, while CuDNNLSTM used ~35% of the GPU, but haven't done a thorough investigation to figure out where the difference comes from.

how are LSTM and CuDNNLSTM different?

This github issue talks about how to convert CuDNNLSTM layers to LSTM layers. I found it pretty illuminating in how CuDNNLSTM has 2x weights/biases as LSTM, and how to convert from one to the other.

$\endgroup$
0
$\begingroup$

Is the 'normal' LSTM assisted by GPU?

Yes, if you have installed tensorflow-gpu.

If so, how are LSTM and CuDNNLSTM different? I presume CuDNNLSTM uses the CUDNN API (and LSTM doesn't? Similarly, is the normal LSTM supposed to be faster running on GPU or CPU?

Have you tried googling:)? There are plenty of links that pop up if you paste this question, such as:

$\endgroup$
  • $\begingroup$ Yes, I have. It's not clear how LSTM and CuDNNLSTM are implemented differently, especially when both LSTM and CuDNNLSTM run on the GPU. $\endgroup$ – John M. Jan 28 '19 at 3:17
0
$\begingroup$

The smallest unit of computation in Tensorflow is called op-kernel. And this op-kernel could be processed from various devices like cpu, gpu, accelerator etc. If the op-kernel was allocated to gpu, the function in gpu library like CUDA, CUDNN, CUBLAS should be called.

Normal Keras LSTM is implemented with several op-kernels. If you use the function like "keras.layers.LSTM(~,implementation=2)", then you will get op-kernel graph with two matmul op-kernels, 1 biasAdd op-kernels, 3 element-wise multiplication op-kernels, and several op-kernels regarding non-linear function and matrix manipulation..

Each of these op-kernels are implemented with independent library and no optimization is applied between these op-kernels. Each of the op-kernels are sorted by the execution order and processed individually in the GPU. (No consideration of optimized processing of multiple op-kernels occurs.)

However, if you use CUDNNLSTM layer in keras, the optimized op-kernel which performs all os the LSTM cell computations at once is created. Compared to normal LSTM layer, it improves the performance(batches/sec) and also the usage of the memory.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.