I've got a regression problem where a model is required to predict a value in the range [0, 1].

I've tried to look at the distribution of the data and and it seems that there are more examples with a low value label ([0, 0.2]) than higher value labels ([0.2, 1]).

When I try to train the model using the MAE metric, the model converges to a state where it has a very low loss, but it seems that the model has converged to a state in which it predicts a low value on many of the high value label examples.

So my assumption was that the data is imbalanced and I should try to weight the loss of the examples depending on their label.

Question: what is the best way to weight the loss in this configuration?

Should I weight each example by the value of its label using some function f(x) , where f(x) is low when x is low and high when x is high?

Or should I split the label values into bins ([0, 0.1), [0.1, 0.2) ... [0.9, 1]) and weight each bin (similarly to categorical loss weight)?

  • $\begingroup$ Do not do it without any statistical look. You can use Pandas' df.describe() to statistically visualize the distribution of your features and labels, then you can actually see what is the mean, variance, min value, max values etc. Then you can decide how to weigh them numerically, and I'd go for weighing the bins as you described at the end. $\endgroup$ – Ugur MULUK Nov 15 '18 at 10:56
  • $\begingroup$ I'd like to ask a few questions regarding you response :) 1. Can you supply some intuition regarding why would you prefer to weigh as bins? I'd love to understand that better 2. What more information would you look for, and how would it influence your weighing? 3. If you suggest to weigh them as bins, than it is the same as categorical weighting. As far as i know in this case you could just weigh them according to the distribution of the bins, Do you agree? if you do, than how would any more statistical information help? :) $\endgroup$ – Gal Avineri Nov 15 '18 at 13:49
  • $\begingroup$ 1) Because it is usually more safe and efficient to use categorical features, rather than numerical. The memory it takes decreases, the algorithms you use will have a speed boost in training (you can use one-hot-encoding to make your dataset even sparser) and categorical features allow us to use the magic of the feature engineering better, in general. 2) I used the word 'safe' at the first sentence; outliers are the danger. If you do not at least statistically manipulate your numerical features, you may end up with strong outliers and with a function f(x), you would reinforce them. $\endgroup$ – Ugur MULUK Nov 16 '18 at 11:09
  • $\begingroup$ 2) (Continued) You can look for the most of the data where your samples fell to, 90% is the rule-of-thumb in general. Let's say 90% of your features are in [0.05, 0.2], the remaining 10% are in the intervals [0, 0.05] and [0.2, 1]. Those remaining samples are the outliers and they could strongly harm your cost (sum of losses) in the training, since they will have large errors yet with a few samples to trust in. My advice is: discretize your values in [0.05, 0.2] as some number of bins, and merge the remaining samples to the bins of [0.05, a] or [b, 0.2] depending on the side. $\endgroup$ – Ugur MULUK Nov 16 '18 at 11:15
  • $\begingroup$ 3)Yes, it is categorical weighting, and yes you can do that, much easily and safer. I agree with you, I think I gave some extra statistical answer at the answer of your question 2. . Hope I could help; if not, do not hesitate to ask. $\endgroup$ – Ugur MULUK Nov 16 '18 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.