Autoencoders for the compression of time series

Question

I am trying to use autoencoder (simple, convolutional, LSTM) to compress time series.

Here are the models I tried.

Simple autoencoder:

    from keras.layers import Input, Dense
    from keras.models import Model
    import keras

    # this is the size of our encoded representations
    encoding_dim = 50

    # this is our input placeholder
    input_ts = Input(shape=(2100,))
    # "encoded" is the encoded representation of the input
    encoded = Dense(encoding_dim, activation='relu')(input_ts) #, activity_regularizer=regularizers.l2(10e-5)
    # "decoded" is the lossy reconstruction of the input
    decoded = Dense(2100, activation='tanh')(encoded)

    # this model maps an input to its reconstruction
    autoencoder = Model(input_ts, decoded)

    # this model maps an input to its encoded representation
    encoder = Model(input_ts, encoded)

    # create a placeholder for an encoded (%encoding_dim%-dimensional) input
    encoded_input = Input(shape=(encoding_dim,))
    # retrieve the last layer of the autoencoder model
    decoder_layer = autoencoder.layers[-1]
    # create the decoder model
    decoder = Model(encoded_input, decoder_layer(encoded_input))

    autoencoder.summary()

    adamax = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.01)
    autoencoder.compile(optimizer=adamax, loss='mean_absolute_percentage_error')

Convolutional autoencoder:

    from keras.layers import Input, Dense, Conv1D, MaxPooling1D, UpSampling1D
    from keras.models import Model

    window_length = 518

    input_ts = Input(shape=(window_length,1))

    x = Conv1D(32, 3, activation="relu", padding="valid")(input_ts)
    x = MaxPooling1D(2, padding="valid")(x)

    x = Conv1D(1, 3, activation="relu", padding="valid")(x)

    encoded = MaxPooling1D(2, padding="valid")(x)

    encoder = Model(input_ts, encoded)

    x = Conv1D(16, 3, activation="relu", padding="valid")(encoded)
    x = UpSampling1D(2)(x) 

    x = Conv1D(32, 3, activation='relu', padding="valid")(x)
    x = UpSampling1D(2)(x)

    decoded = Conv1D(1, 1, activation='tanh', padding='valid')(x)

    convolutional_autoencoder = Model(input_ts, decoded)

    convolutional_autoencoder.summary()

    optimizer = "nadam"
    loss = "mean_absolute_error"

    convolutional_autoencoder.compile(optimizer=optimizer, loss=loss)

LSTM autoencoder:

    from keras.layers import Input, LSTM, RepeatVector
    from keras.models import Model

    inputs = Input(shape=(1, 500))
    encoded = LSTM(128)(inputs)

    decoded = RepeatVector(1)(encoded)

    decoded = LSTM(500, return_sequences=True)(decoded)

    sequence_autoencoder = Model(inputs, decoded)
    encoder = Model(inputs, encoded)

    sequence_autoencoder.summary()

    sequence_autoencoder.compile(optimizer='nadam', loss='mean_absolute_error')

To check for compression loss, I use the SMAPE formula.

I got such results. The average loss for simple autoencoder is 14.28%, for convolutional autoencoder is 8.04%, for LSTM-autoencoder is 9.25%.

My question is: is it practical to compress time series with losses using a neural network if the compression time does not matter? Perhaps i should pay attention to other methods? How can I achieve better compression?

Thanks!

Answer 1

Above all, you should take care of the time series. Despite from that, AEs are thoroughly used for time series, especially LSTM+AE. Did you vary the topology? For the CAE it looks reasonable but the other models lack some layers, or? Furthermore, some regular advices would be to standardize the input, change the activation functions (tanh worked well for me in the output layer) as well as the number of neurons per layer and the amount of layers in general. Could you provide the head() of the input data? An AE expects to fit X on X, maybe you missed that?

Answer 2

I will answer part of your question directly, "Is it practical to compress time series with losses using a neural network if the compression time does not matter".

You try to use an autoencoder to compress data - however, autoencoders without correct regularization methods (or more advanced ideas) are prone to finding simple solutions (you don't put any constraint on the network, so it will very likely converge to the simplest solution it can find that minimizes your loss) that might not generalize well to other data points. I suggest that you split the dataset to train and test and check how well your autoencoder performs on the test set.

If you care about unseen data points (sometimes you don't but in 99% of the use cases you do) the next step, after you probably see that your results are sub-optimal, is to look into something like VAE (variational autoencoders).

All that being said, if you have another method that is not neural, it's mostly safer to stick to that (maybe look into SVD-based approaches, I am not an expert). Neural networks tend to be unpredictable and you probably should not add another failure point to the system unless you have to.

Another plus of "classical" approaches is that it often comes with some theoretical guarantees and information about the amount of variance you get to keep (i.e., "how much" you lose, not what information though, it's "how much" - I find the "what" part often impractical and complicated to interpret). Neural networks are very practical to find a good representation for downstream tasks but I am not sure it's a requirement from your side (and it often requires some method of self-supervision/proxy task, e.g. word embeddings).