I have been working for a while on a regression problem - predicting the air pollution in a city based on meteorological features (humidity, temperature, wind velocity a.o.). I have trained an LSTM model to do the predictions and it works somehow fine, but I am not impressed - the trend in my predictions is always lagging behind the trend in the ground truth. LSTMs are more than 20 years old so I wondered if there are some new alternatives for regression problems of the kind I am working on? I have read that attention models are popular for text prediction, but are they also suitable for air pollution prediction? What else is there? Especially models that are implemented in TensorFlow and Keras would be of interest for me. Thanks!


1 Answer 1


Transformers (aka "attention models") are being used in place of LSTMs in many areas, as they generally give better results, and/or are quicker to train. They can be used for regression problems, just as easily as for classification or text-generation - just create the final layer accordingly.

The catch is they scale as O(N²) in the size of your data. You didn't mention how long the sequences are that you are trying to analyze, but if they are less than 512 items then a transformer should be fine on a single GPU. (And your LSTM is probably doing a poor job with sequences that long anyway.)

There is a lot of active research in making versions of the Transformer that scale to larger values of N. If you read the papers, the claims of O(N) or O(NlogN) are a little misleading as they end up batching things together for efficiency; but still they are managing to work with sequence lengths in the thousands or higher.

For out of the box implementations https://github.com/huggingface/transformers is a good start. For handling larger values of N, I see both Longformer and BigBird mentioned there; the latter was used not just for text but for DNA sequencing. I'm not aware of any research done on meteorological features, specifically, though. (Please leave a comment if you do find anything!)

  • $\begingroup$ Thanks, I am looking now into the transformers! Since you asked - I am using as input sequences of 1 weak of hourly values (7 days *24 hours=168 entries is one sequence long). So do I understand correctly, that the length of the sequences you mention would be in my case 168? I am putting them in batches of 32 or 64. My output are sequences that each are 24 hours long, so sequences of 24 entries. Each of the entries has 10 meteo features (air pollution concentration, temperature, humidity, wind speed...). $\endgroup$
    – NeStack
    Aug 18, 2021 at 8:20
  • $\begingroup$ @NeStack When you said a regression problem I imagined you would want a single number output. To get the 24 predictions, just feed your prediction in as the new input, and run the prediction again. (A lot of the Transformer papers will talk about beam search, but you don't need that extra complexity for a physics problem.) I think "how to turn 10 measurement features into input to a transformer" would be good as a separate question - though I'm guessing you make some kind of embedding already, for use with the LSTM? In which case it should also be usable with a Transformer model. $\endgroup$ Aug 18, 2021 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.