0
$\begingroup$

I have a problem where I take a noisy sequence in (in one domain) and generate a clean sequence out, in another domain. The issue I'm having is that when the input data is relatively clean, the output can be generated with very high numerically accuracy, approximately 1e-6. At some points in some sequences though, the input data is too noisy to do very well and the best possible error may be on the order of 1.0 to 10.0.

My problem is that while training using typical loss functions like MSE, the network is understandably "obsessed" with the high errors, with the result being that at no time does it achieve the very high accuracy obtainable in most cases.

Are there special loss functions or other techniques that can encourage the network to go for highly accurate results when possible without completely punting on the hard cases? I've played around with loss functions that only score the best X% of a batch, but so far it hasn't really helped much, and I still would prefer it to do the best it can on the hard cases.

$\endgroup$
4
  • $\begingroup$ What do you mean with "domain" in this case? $\endgroup$
    – noe
    Apr 27 at 17:45
  • $\begingroup$ It's not really pertinent to the question at hand, but just that my output is not simply a denoised version of input, it also is transformed, y = f(x) in the noiseless case. $\endgroup$
    – Mastiff
    Apr 27 at 17:49
  • $\begingroup$ Are these sequences of variable length or are they of fixed size? $\endgroup$
    – noe
    Apr 27 at 17:58
  • $\begingroup$ Variable. I'm processing with a 1D fully convolutional network. $\endgroup$
    – Mastiff
    Apr 27 at 18:01
1
$\begingroup$

You may try cycle GANs.

In normal Generative Adversarial Networks (GANs), a generative network -the generator- is trained to generate images with an auxiliary network, the discriminator. The discriminator learns to tell apart data generated by the generator (i.e. fake data) from real data, while the generator learns to generate data that fools the discriminator. GANs are known to generate good quality data where MSE and the likes generate "blurry" data that lack detail.

Cycle GANs are two coupled GANs, where the generators learn to "convert" the input data from domain A to domain B, and vice versa, plus an extra loss term to ensure cyclic consistency (i.e. generator1(generator2(a)) = a).

The results are very impressive. Here you can see cycle GANs trained to convert images, changing horses into zebras and vice versa, winter landscape into summer, etc:

enter image description here

If your data is continuous and your architecture is fully convolutional, then cycleGANs may actually work in your case.

$\endgroup$
1
  • $\begingroup$ In the end, did you try cycleGANs? $\endgroup$
    – noe
    Jul 15 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.