Transportation of data to and from the GPU. Data goes into GPU through the bus, allocated into the memory, processed and sends back the results through the same channel. Whether it is noticeable or not it takes time.
Unlike CPU which features sequential execution, GPU features parallel executions and its blazingly fast.
One main disadvantage of GPUs is that it's not built for deep learning instead its originally designed to implement graphics pipelines. As deep learning also align the same kind of computation(matrix multiplications), GPUs were used for deep learning.
More at Are there any disadvantages of using GPU in deep learning?
GPU's are very fast at applying the same instruction to multiple data points (SIMD). However, if you add branching (an if), the two branches will be serialized (first the if on all data that takes it, then the else on all data that doesn't - so still quite parallel, but not entirely). Also, if you have very few data points, the overhead of uploading to the GPU and downloading the result will likely dominate the overall execution time - it might be cheaper to just execute on the CPU instead.