I am testing ideas on IMDB sentiment analysis task by using word embeddings + CNN approach. What could explain a significant difference in computation time in favor of GPU (~9 seconds per epoch) versus TPU (~17 seconds/epoch), despite supposedly superior computational power of a TPU over GPU?
I don't think it makes a lot of sense to compare a generic TPU to a generic GPU. There is probably a factor of 10 or greater between a low-end GPU and the best GPUs on the market in terms of compute capability. Google has itself developed 3 generations of TPUs, each more powerful than the last.
I haven't kept up with the latest developments, but last I was aware, TPUs don't necessarily dominate over GPUs in all aspects. They may provide more compute per watt, but that's not the same as compute per unit or compute per dollar.
More specific to the architecture details, google has it's own documentation on exactly what sorts of computation TPUs are good at, versus what is more suitable for a GPU.