regression with noisy target vairable

Question

How can I approach a regression problem where the input data is not noisy but the target variable is noisy? Are there any regression algorithms that are robust to a noisy target variable?

Also, is it possible to de-noise the target variable somehow? If so, how?

Answer 1

It depends how much noise:

• If it's only a little noise, say for instance 2% of the target values are off by a small value, then you can safely ignore it since the regression method will rely on the most frequent patterns anyway.
• If it's a lot of noise, like 50% of the target values are totally random, then unless you can detect and remove the noisy instances you can forget it: the dataset is useless.

In general ML algorithms are based on statistical principles, to some extent their job is to avoid the noise and focus on the regular patterns. But there are two things to pay attention to:

• Is the noise truly random, or does it introduce some biases in the data? The latter is a much more serious issue.
• Noisy data is even more likely to cause overfitting, so extra precaution should be taken against it: depending on the data, it might be necessary to reduce the number of features and/or the complexity of the model.