2
$\begingroup$

In the case of training a Neural Network on a regression task. Assuming the data has a significant amount of outliers. Provided that the error needs to be RMS and not MAE. Can it be better (as in less sensitive to the outliers) to replace the average over batch size in the weights update by a median over batch size computation?

For a batch size large enough, this should lessen the impact the contribution from the outliers. It does not seem to be common though, at least to current knowledge. What are the shortcomings of this approach?

$\endgroup$
4
  • $\begingroup$ To be more specific, we what to replace the average of weight gradients by median of gradients for each one-dimensional weight? $\endgroup$ – Esmailian Mar 29 '19 at 15:05
  • $\begingroup$ @Esmailian Yes, maybe I was not clear enough, at weight update time the gradients are not averaged in the batch of samples dimension, instead the median over the same axes as the former is taken. $\endgroup$ – Learning is a mess Mar 29 '19 at 15:07
  • $\begingroup$ This could be a breakthrough :) It remotely makes sense. There could be a correspondence between outlier samples and outlier gradients. $\endgroup$ – Esmailian Mar 29 '19 at 15:23
  • $\begingroup$ @Esmailian I am yet to be convinced that this holds a breakthrough. But I am very curious about the cases for which it is more efficient, and how far it can go =) $\endgroup$ – Learning is a mess Apr 1 '19 at 15:25
1
$\begingroup$

One shortcoming is that the median is often more computational expensive to calculate than the mean. The median can be calculated with a variation of quickselect which is linear worst-case performance. Calculating the mean only requires the sum and count of the numbers.

$\endgroup$
1
  • $\begingroup$ There are O(n) algorithms for median computation. $\endgroup$ – Learning is a mess Jan 10 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.