0
$\begingroup$

I'm attempting to develop a recurrent model to forecast the value one step into the future (i.e., $x_{t+1}$), given its history $(x_{t-h},\cdots,x_{t})$, where $h$ is a fixed hyperparameter for the length of histories.

My full data consists of second-level data spanning ~35000 seconds. A sample interval (normalized between 0 and 1) is shown in the plot below. enter image description here

I started with simple models with, say, 1 to 2 layer LSTM with ~50 hidden units each, and parsed the time series with a history of 64 seconds, moving one step ahead for each future samples. The model (implemented in PyTorch is given below)

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim = 1):
        super(LSTMModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_dim = hidden_dim
        self.rnn = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        batch_size, seq_len, dim = x.shape
        #self.hidden = (torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda(), \
        #               torch.zeros(self.num_layers, batch_size, self.hidden_dim).cuda())
        #out, self.hidden = self.rnn(x, self.hidden)
        out, _ = self.rnn(x)
        out = self.fc(out[:,-1].view(batch_size, -1))
        return out.view(-1)

The parts commented out was to test whether a stateful representation is effective. I tested this model with a toy example via learning $x^2/2 + 5x$ and achieved MSE of 0. While training on the dataset of interest, however, I discovered that the model is predicting identical future values (up to 6-point decimals) regardless of my input. I've played with some hyperparameter and optimizer settings but to no avail.

I would really appreciate on any feedback wrt to model building and welcome any suggestions on data processing to help training this model. Thanks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.