0
$\begingroup$

I'm trying to create an LSTM autoencoder to encode some high frequency time series data. The data has 206 time steps and I'm trying to both encode this down to a lower dimension and also recreate the input well. However, the output of my autoencoder often looks like random noise and the reconstruction loss only decreases slightly. I'm kind of lost now and would appreciate any advice.

Here's some code that I'm using. Basically, the encoder takes in data of size (batch_size=16, seq_len=206, input_size=1) and outputs something with dimensions (batch_size=16, hidden=60). The decoder takes in something with dimensions (batch_size=16, hidden=60) and outputs something with dimensions (batch_size=16, seq_len=206).

class LSTMEncoder(torch.nn.Module):
    def __init__(self, hidden):
        super(LSTMEncoder, self).__init__()
        self.input_size = input_size
        self.hidden = hidden

        self.lstm = torch.nn.LSTM(input_size=1, hidden_size=hidden, num_layers=1, batch_first=True)
        self.linear = torch.nn.Linear(in_features=hidden, out_features=hidden)

    def forward(self, x):
        x, (hidden_n, cell_n) = self.lstm(x)
        x = x[:, -1, :]
        x = self.linear(x)
        return x

class LSTMDecoder(torch.nn.Module):
    def __init__(self, hidden):
        super(LSTMDecoder, self).__init__()
        self.hidden = hidden

        self.lstm = torch.nn.LSTM(input_size=1, hidden_size=hidden, num_layers=1, batch_first=True)
        self.linear = torch.nn.Linear(in_features=input_size, out_features=input_size)

    def forward(self, x):
        x = x.unsqueeze(2)
        x, (hidden_n, cell_n) = self.lstm(x)
        x = x[:, -1, :]
        x = self.linear(x)
        return x

class SingleVariableAutoencoder(torch.nn.Module):
    def __init__(self, seq_len, hidden):
        super(SingleVariableAutoencoder, self).__init__()
        self.seq_len = seq_len
        self.hidden = hidden

        self.encoder = LSTMEncoder(hidden)
        self.decoder = LSTMDecoder(seq_len)

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded, encoded

// Training code
autoencoder = SingleVariableAutoencoder(206, 60)
data_load = torch.utils.data.DataLoader(train_ds, len(train_ds), True)
opt = torch.optim.Adam(params=filter(lambda x: x.requires_grad, autoencoder.parameters()), lr=0.01)
sched = torch.optim.lr_scheduler.ExponentialLR(opt, 0.999)
reconstruction_loss_fn = torch.nn.MSELoss()
batch_size = 16

epochs = 1000
for epoch in range(epochs):
    for data_matrix, labels in data_load:
        opt.zero_grad()
        indices = torch.randperm(len(data_matrix))[:batch_size]
        decoder_output, encoder_output = autoencoder(data_matrix[indices].float())
        reconstruction_loss = reconstruction_loss_fn(decoder_output.float(),                 
                                                 data_matrix[indices].float())
        total_loss = (1.)*reconstruction_loss
        total_loss.backward()
        opt.step()
    sched.step()
    print("Epoch: ", epoch, " Total Loss: ", float(total_loss))

And the results look like this, where the blue is the input data and red is the autoencoder output. Sometimes the input and output match very well, but most of the time the output fails to capture the amplitude of the input signals.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.