1
$\begingroup$

Imagine an adaptation of CycleGAN, in which the discriminators were removed in lieu of using only cycle consistency loss. Well, it turns out that the original authors of Cycle Consistent Adversarial Networks attempted this in their paper (figure 7, leftmost generation), and the result was not impressive. I am left wondering; why is that?

Cycle consistency loss requires the output of one generator (G1) to be processed by another generator (G2), and is calculated by computing the difference between this final generated content and the input to G1 (and vice versa). Therefore, it seems that cycle consistency loss alone should be sufficient to enforce the output of both generators to produce output in the desired domain.

Then, why is it necessary for CycleGANs to incorporate discriminators' loss additional to cycle consistency loss? I understand why it is necessary in regular GANs, considering they do not encompass a second generator with which cycle consistency loss can be calculated.

TL;DR: Why do CycleGANs rely on discriminators given that cycle consistency loss alone should - seemingly - be sufficient to enforce meaningful/plausible output?

$\endgroup$
9
  • $\begingroup$ What would be the incentive for generators to learn something different than the identity function? $\endgroup$
    – noe
    Mar 20, 2023 at 15:51
  • $\begingroup$ @noe, touché. If both generators would learn the identity function, then x, G1(x), and G2(G1(x)) would yield the same. However, how does the discriminator incentivize the generation of output in the target domain? All it does is assess whether output is “real” or “fake”; it does not assess whether the output belongs to a certain domain or not. $\endgroup$ Mar 20, 2023 at 22:47
  • $\begingroup$ In CycleGAN, each discriminator considers "real" images only the ones belonging to its domain. Therefore, for a Generator to generate acceptable images, they must belong to the domain accepted by the associated discriminator. I will add an answer with this information. $\endgroup$
    – noe
    Mar 21, 2023 at 7:19
  • $\begingroup$ Please, consider upvoting the answers and accepting one of them. Alternatively, please indicate what is not clear from them. $\endgroup$
    – noe
    Mar 22, 2023 at 9:52
  • $\begingroup$ My bad @noe! Both answers provide a clarification for my question, so I will accept the first one. However, I still do not understand how a discriminator could discern between domain X and domain Y when throughout its training, it has only encountered one of the two domains. $\endgroup$ Mar 22, 2023 at 16:11

2 Answers 2

0
$\begingroup$

Because without the disciminators, there is no objective that enforces the transformed images to follow the target style / distribution.

Explanation

Assume both generators leave the images untouched (i.e. there "generated" image is identical to the input image). This would be a very simple generator with undesired behaviour. The question is: would the objective somehow penalize this behaviour?

Following the notion of the Paper for both generators ($G:X\to Y$, $F:Y\to X$), we then get: $F(y) = y$ and $G(x) = x$. In this case, the cycle consistency loss would be zero (see equation (2) in the paper): $$|F(G(x))-x|_1 = |F(x) - x|_1 = |x - x|_1 = 0$$ and $$|G(F(y))-y|_1 = |G(y) - y|_1 = |y - y|_1 = 0$$

Without discriminators, there is no penalty for $F$ and $G$ not changing the style and this zero-loss-generators would be the optimal solution.

$\endgroup$
6
  • $\begingroup$ thank you! This question is a repetition of an earlier comment: If both generators would learn the identity function, then x, F(x), and G(F(x)) would indeed yield the same result. However, how does the discriminator incentivize the generation of output in the target domain? All it does is assess whether output is “real” or “fake”; it does not assess whether the output belongs to a certain domain or not. $\endgroup$ Mar 20, 2023 at 22:51
  • $\begingroup$ Actually, "real" or "fake" are learned based on given samples from a dataset . In this case, the two discriminators are fed with different datasets. So they learn whether somthing is "a real sample from Domain $X$" or $Y$ respectively. $\endgroup$
    – Broele
    Mar 20, 2023 at 23:16
  • $\begingroup$ I was not aware of the other comment. Must have happened while I was writing my answer. $\endgroup$
    – Broele
    Mar 20, 2023 at 23:19
  • $\begingroup$ I see. Would you not agree, however, that this requires the discriminator to implicitly assess two scenarios: Whether the sample is “real”, and whether the sample is from domain X (or Y)? Therefore, during training, it should encounter four types of samples: Real samples from domain X, fake samples from domain X, real samples from domain Y, and fake samples from Y. However, throughout its training, each discriminator only encounters either the first two types of samples or the last two types of samples. $\endgroup$ Mar 20, 2023 at 23:38
  • $\begingroup$ No. „Real“ is here used as a synonym for „belongs to the domain“ and „fake“ as a synonym for „does not belong to the domain“. More accurate would even be the term „is (not) distinguishable from“. So for the $X$-discriminator everything that looks like $X$ is real and everything else is fake. This discriminator does not know about $Y$. Using these discriminators penalizes if $F(y)$ does not look like it belongs to $X$. $\endgroup$
    – Broele
    Mar 21, 2023 at 0:12
0
$\begingroup$

With only the cycle consistency loss, generators would have no incentive to learn something different than the identity function. This way, if both generators would learn the identity function, then $x$, $G1(x)$, and $G2(G1(x))$ would be the same, meeting the incentives of the cycle consistency loss.

The incentive to learn to generate images of each domain comes from the discriminators. In CycleGAN, each discriminator considers "real" images only the ones belonging to its domain. Therefore, for a Generator to generate acceptable images, they must belong to the domain accepted by the associated discriminator.

$\endgroup$
2
  • $\begingroup$ Thank you! What troubles me is that I do not understand how the discriminators discern between two domains. If they only accept output that is both real and is from one certain domain, then it should have encountered both domains during its training (and this is not the case), correct? $\endgroup$ Mar 21, 2023 at 9:42
  • $\begingroup$ Each discriminator only receives images from one of the two domains. The training process must provide real images from the specific domain to each discriminator. The real images from the two domains are never mixed. $\endgroup$
    – noe
    Mar 21, 2023 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.