I have been working for a while on a regression problem - predicting the air pollution in a city based on meteorological features (humidity, temperature, wind velocity a.o.). I have trained an LSTM model to do the predictions and it works somehow fine, but I am not impressed - the trend in my predictions is always lagging behind the trend in the ground truth. LSTMs are more than 20 years old so I wondered if there are some new alternatives for regression problems of the kind I am working on? I have read that attention models are popular for text prediction, but are they also suitable for air pollution prediction? What else is there? Especially models that are implemented in TensorFlow and Keras would be of interest for me. Thanks!
Transformers (aka "attention models") are being used in place of LSTMs in many areas, as they generally give better results, and/or are quicker to train. They can be used for regression problems, just as easily as for classification or text-generation - just create the final layer accordingly.
The catch is they scale as O(N²) in the size of your data. You didn't mention how long the sequences are that you are trying to analyze, but if they are less than 512 items then a transformer should be fine on a single GPU. (And your LSTM is probably doing a poor job with sequences that long anyway.)
There is a lot of active research in making versions of the Transformer that scale to larger values of
N. If you read the papers, the claims of O(N) or O(NlogN) are a little misleading as they end up batching things together for efficiency; but still they are managing to work with sequence lengths in the thousands or higher.
For out of the box implementations https://github.com/huggingface/transformers is a good start. For handling larger values of N, I see both Longformer and BigBird mentioned there; the latter was used not just for text but for DNA sequencing. I'm not aware of any research done on meteorological features, specifically, though. (Please leave a comment if you do find anything!)