1
$\begingroup$

I have a series of 15 frames with (60 rows x 50 columns). Over the course of those 15 frames, the moon moves from the top left to the bottom right.

Data = https://github.com/aiqc/AIQC/tree/main/remote_datum/image/liberty_moon

enter image description here

enter image description here

enter image description here

I am attempting a walk forward autoencoder where:

  • The input data is a 60x50 image.
  • The evaluation label is a 60x50 image from 2 frames later.
  • All data is scaled between 0-1.
    model = keras.models.Sequential()
    model.add(layers.Conv1D(64*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))
    model.add(layers.Conv1D(32*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))
    model.add(layers.Conv1D(16*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))

    model.add(layers.Conv1D(16*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.UpSampling1D(2))
    model.add(layers.Conv1D(32*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.UpSampling1D(2))
    model.add(layers.Conv1D(64*hp['multiplier'], 3, activation='relu'))
    model.add(layers.UpSampling1D(2))

    model.add(layers.Conv1D(50, 3, activation='sigmoid', padding='same'))
    # last layer tried sigmoid with BCE loss.
    # last layer tried relu with MAE.

Tutorials say to use a final layer of sigmoid and BCE loss, but the values I'm producing must not be between 0-1 because the loss goes way negative.

enter image description here

If I use a final layer of relu with MAE loss it claims to learn something.

enter image description here

But the predicted image is notttt great:

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ Am I just not using enough layers for that amount of pixels? $\endgroup$
    – Kermit
    Commented Jul 31, 2021 at 12:20
  • $\begingroup$ Please explain the experiment. How are you generating a new image? Is the image count=15? $\endgroup$
    – 10xAI
    Commented Aug 8, 2021 at 10:02
  • $\begingroup$ Yes, the count is 15. I screenshotted a video of the moon and cropped them to the exact dimensions using PIL. $\endgroup$
    – Kermit
    Commented Aug 8, 2021 at 10:47
  • 2
    $\begingroup$ Here is exactly how I am running the experiment aiqc.readthedocs.io/en/latest/notebooks/… $\endgroup$
    – Kermit
    Commented Aug 8, 2021 at 16:59

1 Answer 1

Reset to default
0
$\begingroup$

Image auto-encoders are great for style change, but not so great for content change.

Anyways, as you keep ~95% untouched in your transformation, a ResNet would probably be very helpful! If you use an hourglass network, you shall see it quickly converging into a self-encoder (which is ~95% right) and than focus all the training just on the difference (the moon part).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.