2
$\begingroup$

In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less sensitive to the outliers) to replace the average over batch size in the weights update by a median over batch size computation?

For a batch size large enough, this should lessen the impact the contribution from the outliers. It does not seem to be common though, at least to current knowledge. What are the shortcomings of this approach?

$\endgroup$
  • $\begingroup$ To be more specific, we what to replace the average of weight gradients by median of gradients for each one-dimensional weight? $\endgroup$ – Esmailian Mar 29 at 15:05
  • $\begingroup$ @Esmailian Yes, maybe I was not clear enough, at weight update time the gradients are not averaged in the batch of samples dimension, instead the median over the same axes as the former is taken. $\endgroup$ – Learning is a mess Mar 29 at 15:07
  • $\begingroup$ This could be a breakthrough :) It remotely makes sense. There could be a correspondence between outlier samples and outlier gradients. $\endgroup$ – Esmailian Mar 29 at 15:23
  • $\begingroup$ @Esmailian I am yet to be convinced that this holds a breakthrough. But I am very curious about the cases for which it is more efficient, and how far it can go =) $\endgroup$ – Learning is a mess Apr 1 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.