0
$\begingroup$

I've tried a number of experiments with machine learning. From trying to use GANs to upscale images to playing with auto-encoders.

There is one problem that haunts me and always ends up ruining my experiments.

My networks always seem to "learn" to ignore the input and always produce the same image output. This output is often an interesting blend of the input images.

It seems that it's learning to cheat the test and produce the same image regardless of the input.

Here are some examples: enter image description here enter image description here

This has happened in both my GAN and Auto-encoder projects.

How do I avoid this sort of thing happening?

Improve this question
$\endgroup$
4
  • $\begingroup$ You may want to replace ReLU with leaky ReLU. $\endgroup$
    – Green Falcon
    Commented Oct 31, 2020 at 14:50
  • $\begingroup$ @Media why? What How would a Leaky Relu benefit this? I've seen quite a few examples that use Relu. I've always wondered why Leaky Relu exists? $\endgroup$
    – Prime By Design
    Commented Oct 31, 2020 at 18:32
  • 1
    $\begingroup$ There is a problem called dying ReLU. It sometimes happens. $\endgroup$
    – Green Falcon
    Commented Oct 31, 2020 at 19:08
  • $\begingroup$ @Media thanks I'll look into dying relu $\endgroup$
    – Prime By Design
    Commented Nov 1, 2020 at 12:30

1 Answer 1

Reset to default
1
$\begingroup$

Couple of Things straight up:

  1. Inject some noise in the process. When the gan or autoencoder learns that there is some noise it will start to generalise better

  2. Use weaker architectures. (Analogy to weak learners, you cant build random forest wit hstrong trees). Basically allowing for your Architecture to be weak enough to be able to generalise and not learn everything by heart, i.e. map Input to Output.

Improve this answer
$\endgroup$
4
  • $\begingroup$ When you say weak do you mean perhaps have less perceptrons / less layers ? $\endgroup$
    – Prime By Design
    Commented Oct 31, 2020 at 15:12
  • $\begingroup$ Yes, for example that. Or less layers etc... $\endgroup$
    – Noah Weber
    Commented Oct 31, 2020 at 15:21
  • $\begingroup$ So I had a look at the old code, It doesn't really have many output layers at all. $\endgroup$
    – Prime By Design
    Commented Oct 31, 2020 at 18:31
  • $\begingroup$ I had at some point tried to convolve over the image... to try and generalize more. $\endgroup$
    – Prime By Design
    Commented Oct 31, 2020 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.