For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then predict another univariate time series M steps into the future.
I started out by following the "Attention is all you need" paper but since this paper is meant for NLP I had to make some changes. Instead of embedding each word to a random point in d_model-dimensional space I use a linear layer embed the data. I've also tried using a nn.Conv1d layer with a kernel size of 1 to embed, but these approaches fail to make a non-linear prediction of the data and instead only predict a straight line through the average of the data.
First I though that the problem was my implementation of the transformer, but even when I use Pytorch' build in nn.Transformer module I get the same results. I then tried different types of positional encoding like the "Time2Vec" paper that approximates the data by using different sinus functions.
I feel like I've tried a lot of different things to make this transformer work but to no avail. So my question is, do transformers alone work for multistep forecasting with univariate data. And if so are there any articles, papers, repositories etc. that forecasts time series data with succes? If not which approach should I take going forwards to see if I can get my transformer to work.
Edit: I figured out the problem, apparently the only issue was that I had set my learning rate too high :)