1
$\begingroup$

I have a series of 15 frames with (60 rows x 50 columns). Over the course of those 15 frames, the moon moves from the top left to the bottom right.

Data = https://github.com/aiqc/AIQC/tree/main/remote_datum/image/liberty_moon

enter image description here

enter image description here

enter image description here

I am attempting a walk forward autoencoder where:

  • The input data is a 60x50 image.
  • The evaluation label is a 60x50 image from 2 frames later.
  • All data is scaled between 0-1.
    model = keras.models.Sequential()
    model.add(layers.Conv1D(64*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))
    model.add(layers.Conv1D(32*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))
    model.add(layers.Conv1D(16*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.MaxPool1D( 2, padding='same'))

    model.add(layers.Conv1D(16*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.UpSampling1D(2))
    model.add(layers.Conv1D(32*hp['multiplier'], 3, activation='relu', padding='same'))
    model.add(layers.UpSampling1D(2))
    model.add(layers.Conv1D(64*hp['multiplier'], 3, activation='relu'))
    model.add(layers.UpSampling1D(2))

    model.add(layers.Conv1D(50, 3, activation='sigmoid', padding='same'))
    # last layer tried sigmoid with BCE loss.
    # last layer tried relu with MAE.

Tutorials say to use a final layer of sigmoid and BCE loss, but the values I'm producing must not be between 0-1 because the loss goes way negative.

enter image description here

If I use a final layer of relu with MAE loss it claims to learn something.

enter image description here

But the predicted image is notttt great:

enter image description here

$\endgroup$
4
  • $\begingroup$ Am I just not using enough layers for that amount of pixels? $\endgroup$
    – Kalanos
    Jul 31 at 12:20
  • $\begingroup$ Please explain the experiment. How are you generating a new image? Is the image count=15? $\endgroup$
    – 10xAI
    Aug 8 at 10:02
  • $\begingroup$ Yes, the count is 15. I screenshotted a video of the moon and cropped them to the exact dimensions using PIL. $\endgroup$
    – Kalanos
    Aug 8 at 10:47
  • 2
    $\begingroup$ Here is exactly how I am running the experiment aiqc.readthedocs.io/en/latest/notebooks/… $\endgroup$
    – Kalanos
    Aug 8 at 16:59
0
$\begingroup$

Image auto-encoders are great for style change, but not so great for content change.

Anyways, as you keep ~95% untouched in your transformation, a ResNet would probably be very helpful! If you use an hourglass network, you shall see it quickly converging into a self-encoder (which is ~95% right) and than focus all the training just on the difference (the moon part).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.