2
$\begingroup$

How does one-sided label smoothing make the discriminator more robust by reducing the confidence in correct class?

$\endgroup$
2
$\begingroup$

Well, let's start with what label smoothing is. We replace the 0s and 1s in one hot encoding for example for the MNIST dataset with 10 classes by smoothed values e.g.: 0.1 for 1, 0.9 for 9 etc.

If we replace positive classification targets with $\alpha$ and negative targets with $\beta$ the optimal discriminator $D$ becomes:

$$D(x)=\frac{\alpha p_{data}(x) + \beta p_{model}(x)}{p_{data}(x) + p_{model}(x)}$$.

Where the $p_{data}(x)$ corresponds to the original distribution; at least we think. And the $p_{model}(x)$ corresponds to the distribution that we want to target and $x$ are the input features to the discriminator $D$.

Now, when $\beta \neq 0$, erroneous samples from $p_{model}$ have no incentive to move nearer to the data i.e. it reinforces current generator behaviour. Hence, we smooth only the positive labels to $\alpha$, leaving negative labels set to 0. This behaviour prevents discriminator from giving very large gradient signal to the generator.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.