Placing too much probability mass on a single $x$ is indeed a major problem with GANs. Usually it is referred to as mode collapse. Most GAN papers will discuss it.
I was a bit confused as to how training 𝐺 for a shorter amount of time before running 𝐷 can fix it.
The idea is to keep the discriminator $D$ up to date with the generator $G$. By training $G$ less, you are training $D$ more. Usually, an alternating optimization is being done with GANs, i.e. train $D$ for $n$ iterations, then $G$ for $m$ iterations, and so on. The "job" of $D$ is to be able to discern fake $G$ outputs from real data points. But, if you don't train $D$ enough (i.e., you train $G$ too much, so $m>n$), then $G$ will quickly "run off" and exploit local minima in $D$. From $G$'s point of view, if $D$ "likes" some output $x=G(z)$, then the easiest thing to do is just only output that $x$! Training $G$ less (and thus $D$ more) allows this minima to move, and prevents $G$ from exploiting them.
Notice that overtraining $D$ doesn't seem to be a bad thing a priori.
Indeed, with more modern GAN losses (e.g., Wasserstein ones), one wants to $n$ to be as large as possible.
It just means that $D$ converges to being a great discriminator at every step, so it should be informative for $G$. (Indeed, proofs of convergence hinge on doing this!) But in practice, for the original GAN, overtraining $D$ tends to lead to useless gradients (see below): nothing $G$ can do will please the discriminator, so all weight updates are useless, so the GAN just futilely flails around parameter space). This flailing is called unstable training by others. :)
In other words, for the vanilla GAN, undertraining $D$ can lead to mode collapse, while overtraining it can cause instability. People seemingly have focused on mitigating instability so that one simply overtrains $D$ and avoids mode collapse.
how it can be prevented?
One common way to mitigate it is to use a different loss function, such as a Wasserstein loss. One problem with the original GAN formulation is that (by minimizing the JS divergence) can have very uninformative gradients because the loss does not handle distributions with very little overlap very well (i.e., when the shared support is small). In some sense, it is too harsh when the generator is doing poorly. In contrast, an optimal transport loss smoothly decreases even when the the two distributions are far from each other. See e.g. the Wasserstein GAN paper and its followups.
And why is it called "the Helvetica scenario"?
It refers to the show you mentioned, where too much Calcium is put in one place and causes a disaster. Here, a disaster occurs when too much generator probability density is placed in a small area of the data space. :)
