Techniques to manage high dynamic range in neural network regression?

Question

I have a problem where I take a noisy sequence in (in one domain) and generate a clean sequence out, in another domain. The issue I'm having is that when the input data is relatively clean, the output can be generated with very high numerically accuracy, approximately 1e-6. At some points in some sequences though, the input data is too noisy to do very well and the best possible error may be on the order of 1.0 to 10.0.

My problem is that while training using typical loss functions like MSE, the network is understandably "obsessed" with the high errors, with the result being that at no time does it achieve the very high accuracy obtainable in most cases.

Are there special loss functions or other techniques that can encourage the network to go for highly accurate results when possible without completely punting on the hard cases? I've played around with loss functions that only score the best X% of a batch, but so far it hasn't really helped much, and I still would prefer it to do the best it can on the hard cases.

Answer 1

You may try cycle GANs.

In normal Generative Adversarial Networks (GANs), a generative network -the generator- is trained to generate images with an auxiliary network, the discriminator. The discriminator learns to tell apart data generated by the generator (i.e. fake data) from real data, while the generator learns to generate data that fools the discriminator. GANs are known to generate good quality data where MSE and the likes generate "blurry" data that lack detail.

Cycle GANs are two coupled GANs, where the generators learn to "convert" the input data from domain A to domain B, and vice versa, plus an extra loss term to ensure cyclic consistency (i.e. generator1(generator2(a)) = a).

The results are very impressive. Here you can see cycle GANs trained to convert images, changing horses into zebras and vice versa, winter landscape into summer, etc:

If your data is continuous and your architecture is fully convolutional, then cycleGANs may actually work in your case.