I'm attempting to develop a recurrent model to forecast the value one step into the future (i.e., $x_{t+1}$), given its history $(x_{t-h},\cdots,x_{t})$, where $h$ is a fixed hyperparameter for the length of histories.

My full data consists of second-level data spanning ~35000 seconds. A sample interval (normalized between 0 and 1) is shown in the plot below. enter image description here

I started with simple models with, say, 1 to 2 layer LSTM with ~50 hidden units each, and parsed the time series with a history of 64 seconds, moving one step ahead for each future samples. The model (implemented in PyTorch is given below)

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim = 1):
        super(LSTMModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_dim = hidden_dim
        self.rnn = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        batch_size, seq_len, dim = x.shape
        #self.hidden = (torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda(), \
        #               torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda())
        #out, self.hidden = self.rnn(x, self.hidden)
        out, _ = self.rnn(x)
        out = self.fc(out[:,-1].view(batch_size, -1))
        return out.view(-1)

The parts commented out was to test whether a stateful representation is effective. I tested this model with a toy example via learning $x^2/2 + 5x$ and achieved MSE of 0. While training on the dataset of interest, however, I discovered that the model is predicting identical future values (up to 6-point decimals) regardless of my input. I've played with some hyperparameter and optimizer settings but to no avail.

I would really appreciate on any feedback wrt to model building and welcome any suggestions on data processing to help training this model. Thanks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.