I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). Therefor, the model seems to output just values less than 2 (if the range is [0,6]), for instance, and it absolutely ignores bigger target's values, which have a smaller performance in the dataset. How is it possible to improve the model's results in such a case?

For example, when it comes to a multiclass classification, we can weigh penalties for errors on a smaller class to enhance performance with imbalanced data. Are there any tricks in terms of regression? Which loss-functions could be useful? It seems, that MSE loss function is more preferable than RMSE. Is it more powerful loss functions for this problem?

There is a paper about such a problem of imbalanced regression (http://proceedings.mlr.press/v74/branco17a/branco17a.pdf) which might be helpful to someone. However, I'm more interested in special tricks for a neural network, not pre-processing approaches (I can't generate more data for example).

Picture is just an example of distribution

  • $\begingroup$ Hi, Lana! Could you be more specific in regards to how your dataset is structured? The specifications of this exponential distribution could be useful, or at all what specifically you are trying to achieve? Is it exponential regression? I'm also confused how this is multiclass? $\endgroup$ – Andreas Storvik Strauman Jul 26 '19 at 16:02
  • $\begingroup$ @AndreasStorvikStrauman I'm trying to enhance performance on instances, which have a smaller representation in dataset, because now the model ignores such cases. The references to a multiclass classification is just an example, how a similar problem could be solved not for regression. It's not an exponential regression. $\endgroup$ – Lana Jul 26 '19 at 20:10
  • $\begingroup$ It could be useful to make a classification first. First, you classify data in larger values or lower values and them you construct a regression model for each one. In this way, you can be more accurate for larger values. $\endgroup$ – Skullgreymon Jul 27 '19 at 9:53
  • $\begingroup$ @Skullgreymon thank you for your comment! It's really reasonable decision to consider a classification problem first. However, I'm interested to figure out especially, how it could be enhance in terms of regression. $\endgroup$ – Lana Jul 27 '19 at 10:04
  • $\begingroup$ Well, since you are using a Neural Network to do regression, I don't see how loss weight balancing would be a part of preprocessing? Did you try e.g. weighted batch sampling during for stochastic gradient descent? $\endgroup$ – Andreas Storvik Strauman Jul 27 '19 at 12:09

Neural Networks can in general be interpreted as a regression problem and as such, you could apply well known ways of dealing with this. This paper gives you a good introduction to different approaches. For instance you can upsample the minority class, or you could do loss weight balancing during training.

For instance, consider the data point $x_i$ that can belong to one of two classes $a$ and $b$. For instance, class $a$ is here the minority. Then you would, during training, multiply the calculated loss with the weight. $$L_i=\begin{cases}l_a\cdot x_i,&\text{if datapoint $x_i$ is in the minority class}\\l_b\cdot x_i,&\text{if datapoint $x_i$ is in the majority class}\end{cases}$$ with $l_a$ > $l_b$ which would be natural for this example.

You could also look into continous performance measures that could work nicely with imbalanced dataset. For instance the generalized dice (F1) score.

  • $\begingroup$ @Lana did you not get it to work? $\endgroup$ – Andreas Storvik Strauman Jul 28 '19 at 8:48
  • $\begingroup$ Thank you for your answer, but does generalize dice loss work in regression task, if it evaluates multiple class segmentation loss (maybe I missunderstood something)? Although I don't see any mentions of regression task in the paper you have attached as an introduction, it's a useful material. $\endgroup$ – Lana Jul 29 '19 at 12:45
  • $\begingroup$ @Lana Well, I believe generalized dice only works for a two-class problem. Are you performing the regression differently from how you would train a normal supervised neural network (with soft classification)? If so, please specify how you are approaching your regression problem! If not, then most of the approaches described in the imbalanced-dataset paper, as well as weight balancing mentioned in the answer, should work just fine :) $\endgroup$ – Andreas Storvik Strauman Jul 31 '19 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.