2
$\begingroup$

I've got a regression problem where a model is required to predict a value in the range [0, 1].

I've tried to look at the distribution of the data and and it seems that there are more examples with a low value label ([0, 0.2]) than higher value labels ([0.2, 1]).

When I try to train the model using the MAE metric, the model converges to a state where it has a very low loss, but it seems that the model has converged to a state in which it predicts a low value on many of the high value label examples.

So my assumption was that the data is imbalanced and I should try to weight the loss of the examples depending on their label.

Question: what is the best way to weight the loss in this configuration?

Should I weight each example by the value of its label using some function f(x) , where f(x) is low when x is low and high when x is high?

Or should I split the label values into bins ([0, 0.1), [0.1, 0.2) ... [0.9, 1]) and weight each bin (similarly to categorical loss weight)?

$\endgroup$
9
  • $\begingroup$ Do not do it without any statistical look. You can use Pandas' df.describe() to statistically visualize the distribution of your features and labels, then you can actually see what is the mean, variance, min value, max values etc. Then you can decide how to weigh them numerically, and I'd go for weighing the bins as you described at the end. $\endgroup$
    – Ugur MULUK
    Commented Nov 15, 2018 at 10:56
  • $\begingroup$ I'd like to ask a few questions regarding you response :) 1. Can you supply some intuition regarding why would you prefer to weigh as bins? I'd love to understand that better 2. What more information would you look for, and how would it influence your weighing? 3. If you suggest to weigh them as bins, than it is the same as categorical weighting. As far as i know in this case you could just weigh them according to the distribution of the bins, Do you agree? if you do, than how would any more statistical information help? :) $\endgroup$ Commented Nov 15, 2018 at 13:49
  • $\begingroup$ 1) Because it is usually more safe and efficient to use categorical features, rather than numerical. The memory it takes decreases, the algorithms you use will have a speed boost in training (you can use one-hot-encoding to make your dataset even sparser) and categorical features allow us to use the magic of the feature engineering better, in general. 2) I used the word 'safe' at the first sentence; outliers are the danger. If you do not at least statistically manipulate your numerical features, you may end up with strong outliers and with a function f(x), you would reinforce them. $\endgroup$
    – Ugur MULUK
    Commented Nov 16, 2018 at 11:09
  • $\begingroup$ 2) (Continued) You can look for the most of the data where your samples fell to, 90% is the rule-of-thumb in general. Let's say 90% of your features are in [0.05, 0.2], the remaining 10% are in the intervals [0, 0.05] and [0.2, 1]. Those remaining samples are the outliers and they could strongly harm your cost (sum of losses) in the training, since they will have large errors yet with a few samples to trust in. My advice is: discretize your values in [0.05, 0.2] as some number of bins, and merge the remaining samples to the bins of [0.05, a] or [b, 0.2] depending on the side. $\endgroup$
    – Ugur MULUK
    Commented Nov 16, 2018 at 11:15
  • $\begingroup$ 3)Yes, it is categorical weighting, and yes you can do that, much easily and safer. I agree with you, I think I gave some extra statistical answer at the answer of your question 2. . Hope I could help; if not, do not hesitate to ask. $\endgroup$
    – Ugur MULUK
    Commented Nov 16, 2018 at 11:16

3 Answers 3

1
$\begingroup$

If you are predicting values between 0 and 1, you should use beta regression.

Beta regression will handle the heteroskedasticity or skewness which are commonly observed in rates or proportions.

$\endgroup$
1
$\begingroup$

This behavior seems completely reasonable to me and raises no red flags.

You know that most of the values are small. Consequently, the likelihood of getting a large value is small, even when you consider the features.

Your predictions align with this: predictions tend to be small, as I would argue they should be.

There might be room for improvement. One answer mentions beta regression, which could be an option to explore. If your values are bounded between zero and one because they are proportions of times events happened, you can consider obtaining the original counts and modeling the discrete outcomes.

The key point, however, is that this model seems to be doing what it should be doing.

$\endgroup$
1
$\begingroup$

To handle imbalanced data in regression tasks, use the DenseWeights package. It assigns weights to data points using kernel density estimation (KDE) based on the rarity of their target values. For a visual demonstration, see the figure from the associated paper below.

The best part about this method is that they wrapped their work in a Python package - [Link]. So, we can directly use this in our loss function.

The method was introduced in a paper titled "Density-based weighting for imbalanced regression" by Steininger, Michael, et al. You can find more information about this method in the paper - [Link]

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.