Is time series forecasting possible with a transformer?

Question

For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then predict another univariate time series M steps into the future.

I started out by following the "Attention is all you need" paper but since this paper is meant for NLP I had to make some changes. Instead of embedding each word to a random point in d_model-dimensional space I use a linear layer embed the data. I've also tried using a nn.Conv1d layer with a kernel size of 1 to embed, but these approaches fail to make a non-linear prediction of the data and instead only predict a straight line through the average of the data.

First I though that the problem was my implementation of the transformer, but even when I use Pytorch' build in nn.Transformer module I get the same results. I then tried different types of positional encoding like the "Time2Vec" paper that approximates the data by using different sinus functions.

I feel like I've tried a lot of different things to make this transformer work but to no avail. So my question is, do transformers alone work for multistep forecasting with univariate data. And if so are there any articles, papers, repositories etc. that forecasts time series data with succes? If not which approach should I take going forwards to see if I can get my transformer to work.

Edit: I figured out the problem, apparently the only issue was that I had set my learning rate too high :)

Answer 1

Transformers can be used for time series forecasting. See the following articles:

• Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al.
• Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case, by Neo Wu, Bradley Green, Xue Ben, & Shawn O'Banion
• The Time Series Transformer, by Theodoras Ntakouris
• Transformers for Time-series Forecasting, by Natasha Klingebrunn_

Answer 2

I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else.

With time series you aren't changing something to something else, you're trying to find statistical patterns in variations across time e.g do these variations correlate with each other, do they follow trends, cycles, etc. Then you're trying to use these statistical patterns to forecast for future dates. Time series is a lot like regression in that sense, where transformers try to capture relationships between words, and not just in left to right direction (the positional encoding is just extra information for the network).

Answer 3

Transformers were originally architected for NLP. However many studies have shown that they CAN be used for time series as well and with great success.

Let us look at the differences and similarities and the industry developments around that particular area. Once the above points are understood, refining your transformer model to make it work for time-series should be easier.

• Unlike in NLP, where attention matrices revolve around the neighbouring word, here the attention matrices would stretch way back into the past to factor in trends like seasonality. This of course means ability to handle longer sequences is the first key to making transformers work for time-series. Enter many interesting models which tweak the self-attention process of Vaswani.

• Handling long sequences is one aspect, but there is a side-effect. Due to Softmax activation used in transformers, there are too many non-zero attention weights now. This could dilute the real weights. The very usage of Softmax now becomes questionable. Enter α-entmax, a simple layer which replaces Softmax. No more diluted focus with 1000's of time steps to distract the attention mechanism. If necessary, the attention mechanism can zoom in on (ie assign 100% weightage to) one single token 500 time steps back. Various Sparse attention models are now the trend in transformers (both for NLP and time series). At least a dozen good papers have been released in 2020 on such models. Linformer is especially focussed on time-series

• Temporal nature of data is the essence here. This is way different from translating a sentence - an activity which does not have too many temporal patterns. We could also have multi-variate problems. Along with the main time-series, there are moving averages and all sorts of other data which can be considered for modelling. https://arxiv.org/abs/2010.02803 is a good place to start

• Does a 1-1 dot product make sense for a time-series? What is the intuition behind a dot product of 2 numbers? Wouldn't it be more useful to slightly widen the horizon and take a dot product across a few time steps(Blocks of words dot product with blocks of words). It turns out this too is true and several new papers are now leveraging a mix of convolution and traditional self-attention to create interesting new approaches

• Are all parts of the NLP transformer relevant to a time-series? In particular - Norm - has been known to create problems when used with regression

• Predicting single steps versus predicting larger time horizons which are the need in time-series. Of course this means that teacher forcing is not an option and errors could accumulate. Again Informer is a very good option - https://arxiv.org/pdf/2012.07436.pdf

• There is no Wikipedia to train a time-series, but an interesting experiment could be to take all the stock prices in the world over last century (all public data) and use MLM-like objective to create a TS-RoBERTa. Maybe probes would show that this model clusters average TS embeddings according to their mathematical distributions & temporal properties. Maybe we can even use transfer learning onto small training sets?

To sum it up, transformers can and should be evaluated for time series problems. Very often they work without any major architectural changes. Sometimes, some components do need changes considering the inherent differences between NLP and time-series data. A plethora of transformer models have been released in 2020 which will suit time-series rather well. Linformer is a good option to start...