I am expanding my knowledge of the Keras package and I have been tooling with some of the available models. I have an NLP binary classification problem that I'm trying to solve and have been applying different models.
After working with some results and reading more and more about LSTM, it seems like this approach is far superior to anything else I've tried (across multiple datasets). I keep thinking to myself, "why/when would you not use LSTM?". The use of the additional gates, inherent to LSTM, makes perfect sense to me after having some models that suffer from vanishing gradients.
So what's the catch with LSTM? Where do they not do so well? I know there is no such thing as a "one size fits all" algorithm, so there must be a downside to LSTM.