The smallest unit of computation in Tensorflow is called op-kernel. And this op-kernel could be processed from various devices like cpu, gpu, accelerator etc. If the op-kernel was allocated to gpu, the function in gpu library like CUDA, CUDNN, CUBLAS should be called.
Normal Keras LSTM is implemented with several op-kernels. If you use the function like "keras.layers.LSTM(~,implementation=2)", then you will get op-kernel graph with two matmul op-kernels, 1 biasAdd op-kernels, 3 element-wise multiplication op-kernels, and several op-kernels regarding non-linear function and matrix manipulation..
Each of these op-kernels are implemented with independent library and no optimization is applied between these op-kernels. Each of the op-kernels are sorted by the execution order and processed individually in the GPU. (No consideration of optimized processing of multiple op-kernels occurs.)
However, if you use CUDNNLSTM layer in keras, the optimized op-kernel which performs all os the LSTM cell computations at once is created. Compared to normal LSTM layer, it improves the performance(batches/sec) and also the usage of the memory.