You are right that LSTMs work very well for some problems, but some of the drawbacks are:

 - LSTMs take longer to train
 - LSTMs require more memory to train
 - LSTMs are easy to overfit
 - Dropout is much harder to implement in LSTMs
 - LSTMs are sensitive to different random weight initializations

These are in comparison to a different model like 1D conv nets, for example.

The first three items are because LSTMs have more parameters.