For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then predict another univariate time series M steps into the future.

I started out by following the "Attention is all you need" paper but since this paper is meant for NLP I had to make some changes. Instead of embedding each word to a random point in d_model-dimensional space I use a linear layer embed the data. I've also tried using a nn.Conv1d layer with a kernel size of 1 to embed, but these approaches fail to make a non-linear prediction of the data and instead only predict a straight line through the average of the data.

First I though that the problem was my implementation of the transformer, but even when I use Pytorch' build in nn.Transformer module I get the same results. I then tried different types of positional encoding like the "Time2Vec" paper that approximates the data by using different sinus functions.

I feel like I've tried a lot of different things to make this transformer work but to no avail. So my question is, do transformers alone work for multistep forecasting with univariate data. And if so are there any articles, papers, repositories etc. that forecasts time series data with succes? If not which approach should I take going forwards to see if I can get my transformer to work.


I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else.

With time series you aren't changing something to something else, you're trying to find statistical patterns in variations across time e.g do these variations correlate with each other, do they follow trends, cycles, etc. Then you're trying to use these statistical patterns to forecast for future dates. Time series is a lot like regression in that sense, where transformers try to capture relationships between words, and not just in left to right direction (the positional encoding is just extra information for the network).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.