2
$\begingroup$

I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen data?

$\endgroup$
  • $\begingroup$ no I don‘t think this is a good approach. You should try to change your model, e.g. by regularization. $\endgroup$ – Peter Nov 9 '19 at 21:09
3
$\begingroup$

Yes, adding noise can help to regularize a model.

It is well known that the addition of noise to the input data of a neural network during training can, in some circumstances, lead to significant improvements in generalization performance

from Training with Noise is Equivalent to Tikhonov Regularization

In particular, adding structured noise that is consistent with natural perturbations of the data could help regularized a model. This is a form of data augmentation.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I guess I agree with both Answers, and I may agree partially with @Dan Carter Below. Dan's answer makes sense if and only if the training and the validation dataset differs significantly (e.g., training a model to detect emotions where input faces belong to one ethnicity, (German for example), then adding noise to these input faces won't help the model generalize to Chinese input faces. Therefore, if the distribution of both training and the validation dataset are close, then I would assume that data augmentation works. Can Brain and Dan please comment on this observation?Thank you $\endgroup$ – I. A Nov 10 '19 at 21:26
  • $\begingroup$ Maybe augmenting German faces in an attempt to help a model generalize to French faces is useful (This is where augmentation can help) $\endgroup$ – I. A Nov 10 '19 at 21:33
3
$\begingroup$

No, adding noise will not help to regularise the model. This won't help your model generalise better to unseen inputs. It will probably just make your model perform generally worse. Any modifications you make to the training instances should be learnable.

Take an image classification problem as an example (images of a cat or not cat): randomly converting pixels to white in every image will not help your model learn generalisable features of cats; random white noise is not a 'learnable' feature. However by rotating, flipping, cropping, or adjusting contrast of the images, and adding these as new training instances, this would force the model to be more tolerant of differences in each image of cats, and therefore generalise better.

There are many methods for regularising models, such as l1 or l2 regularisation in linear regression or neural nets, and also plenty of other methods specific to other model types.

Making sure your training data is as varied as possible, and using model specific regularisation techniques, are probably closer to 'best practise' (though I don't really like the term since 'best practises' often change, such as new regularisation techniques, optimisations etc etc).

Update:
I've since experimented with adding Gaussian noise to input features, and especially in cases with overfit neural nets this can help the model generalise. It's also possible to add GaussianNoise layers directly in a neural net for this purpose, as in the TensorFlow documentation here.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.