# Time Series Forecasting with RNNs

I'm attempting to develop a recurrent model to forecast the value one step into the future (i.e., $$x_{t+1}$$), given its history $$(x_{t-h},\cdots,x_{t})$$, where $$h$$ is a fixed hyperparameter for the length of histories.

My full data consists of second-level data spanning ~35000 seconds. A sample interval (normalized between 0 and 1) is shown in the plot below.

I started with simple models with, say, 1 to 2 layer LSTM with ~50 hidden units each, and parsed the time series with a history of 64 seconds, moving one step ahead for each future samples. The model (implemented in PyTorch is given below)

class LSTMModel(nn.Module):
def __init__(self, input_dim, hidden_dim, num_layers, output_dim = 1):
super(LSTMModel, self).__init__()
self.num_layers = num_layers
self.hidden_dim = hidden_dim
self.rnn = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
self.fc = nn.Linear(hidden_dim, output_dim)

def forward(self, x):
batch_size, seq_len, dim = x.shape
#self.hidden = (torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda(), \
#               torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda())
#out, self.hidden = self.rnn(x, self.hidden)
out, _ = self.rnn(x)
out = self.fc(out[:,-1].view(batch_size, -1))
return out.view(-1)


The parts commented out was to test whether a stateful representation is effective. I tested this model with a toy example via learning $$x^2/2 + 5x$$ and achieved MSE of 0. While training on the dataset of interest, however, I discovered that the model is predicting identical future values (up to 6-point decimals) regardless of my input. I've played with some hyperparameter and optimizer settings but to no avail.

I would really appreciate on any feedback wrt to model building and welcome any suggestions on data processing to help training this model. Thanks.