I have a problem where I take a noisy sequence in (in one domain) and generate a clean sequence out, in another domain. The issue I'm having is that when the input data is relatively clean, the output can be generated with very high numerically accuracy, approximately 1e-6. At some points in some sequences though, the input data is too noisy to do very well and the best possible error may be on the order of 1.0 to 10.0.
My problem is that while training using typical loss functions like MSE, the network is understandably "obsessed" with the high errors, with the result being that at no time does it achieve the very high accuracy obtainable in most cases.
Are there special loss functions or other techniques that can encourage the network to go for highly accurate results when possible without completely punting on the hard cases? I've played around with loss functions that only score the best X% of a batch, but so far it hasn't really helped much, and I still would prefer it to do the best it can on the hard cases.