2
$\begingroup$

I just built a Convolutional Autoencoder to try to reconstruct a time series with shape (4000, 10, 30). This is the code, I used a batch size of 32, but I think it is overfitting since it performs well on the training set but starts not reconstruction well in the validation and test set. What are the steps I can do to improve it? How can I define the right number of filters for each layer?

kernel_size = 7
stride = 1
model = Sequential()
model.add(Conv1D(filters=128, kernel_size=kernel_size, activation='relu', padding="same", strides=stride,
                 input_shape=(TIME_STEPS, n_features)))
model.add(Conv1D(filters=64, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1D(filters=32, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=16, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=16, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(UpSampling1D(size=2))
model.add(Conv1DTranspose(filters=32, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=64, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=128, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=n_features, kernel_size=kernel_size, padding="same"))

This is the optimizer:

optimizer = tf.keras.optimizers.Adam(learning_rate=1e-04)
model.compile(optimizer=optimizer, loss="mse",
              metrics=[tf.metrics.MeanAbsoluteError(), 'accuracy'])

EDIT: Dropout version:

model = Sequential()
model.add(Conv1D(filters=20, kernel_size=kernel_size, activation='relu', padding="same", strides=stride,
                 input_shape=(TIME_STEPS, n_features)))
model.add(Dropout(0.2))
model.add(Conv1D(filters=15, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(MaxPooling1D(pool_size=2))
model.add(Conv1D(filters=10, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=10, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(UpSampling1D(size=2))
model.add(Conv1DTranspose(filters=15, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Dropout(0.2))
model.add(Conv1DTranspose(filters=20, kernel_size=kernel_size, activation='relu', padding="same", strides=stride))
model.add(Conv1DTranspose(filters=n_features, kernel_size=kernel_size, padding="same"))
Improve this question
$\endgroup$
4
  • $\begingroup$ Could you clarify if the question is "how to not overfit?" vs. "how to get better accuracy?". The former can have the canonical answer you want (i.e. regularization), while the latter might not. (Try building a baseline model, so you have something to compare against.) $\endgroup$
    – Darren Cook
    Commented Apr 6, 2021 at 9:03
  • 1
    $\begingroup$ @DarrenCook My problem is that the model has high accuracy for the 80% of the training set, then it starts performing worse on the last part of the train set and in almost all the validation and test samples. I build a ConvLSTM which performs better (90% accuracy) but I would prefer to build a Convolutional Autoencoder like the one I posted. This is the ConvLSTM:pastebin.com/7vN98f7D $\endgroup$
    – Fabio
    Commented Apr 6, 2021 at 9:12
  • $\begingroup$ As a time series, train comes before valid comes before test in time order? Could there have been some kind of shift in the data in the last part of the training data? (E.g. a global pandemic, or a stock market crash.) Though if your ConvLSTM is not affected by it, that is strange/interesting. $\endgroup$
    – Darren Cook
    Commented Apr 6, 2021 at 9:17
  • $\begingroup$ @DarrenCook yes they are ordered as train-val-test and I did't shuffle them. I also tried shuffling train and val but nothing changed. $\endgroup$
    – Fabio
    Commented Apr 6, 2021 at 9:40

1 Answer 1

Reset to default
2
+50
$\begingroup$

After taking a look at your code, it seems that you've not employed any kind of regularization. You may want to use dropout. Moreover, in convolutional autoencoders, in the decoder part, there is a well-known artifact called checkerboard. I don't know how this can be a problem for your task since you're using one-dimensional convolution in the decoder. By the way, I guess using dropout will suffice. Try to use both in decoder and encoder.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – oW_
    Commented Apr 6, 2021 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.