I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). Therefor, the model seems to output just values less than 2 (if the range is [0,6]), for instance, and it absolutely ignores bigger target's values, which have a smaller performance in the dataset. How is it possible to improve the model's results in such a case?

For example, when it comes to a multiclass classification, we can weigh penalties for errors on a smaller class to enhance performance with imbalanced data. Are there any tricks in terms of regression? Which loss-functions could be useful? It seems, that MSE loss function is more preferable than RMSE. Is it more powerful loss functions for this problem?

There is a paper about such a problem of imbalanced regression (http://proceedings.mlr.press/v74/branco17a/branco17a.pdf) which might be helpful to someone. However, I'm more interested in special tricks for a neural network, not pre-processing approaches (I can't generate more data for example).

Picture is just an example of distribution

  • $\begingroup$ Hi, Lana! Could you be more specific in regards to how your dataset is structured? The specifications of this exponential distribution could be useful, or at all what specifically you are trying to achieve? Is it exponential regression? I'm also confused how this is multiclass? $\endgroup$ Jul 26, 2019 at 16:02
  • $\begingroup$ @AndreasStorvikStrauman I'm trying to enhance performance on instances, which have a smaller representation in dataset, because now the model ignores such cases. The references to a multiclass classification is just an example, how a similar problem could be solved not for regression. It's not an exponential regression. $\endgroup$
    – Lana
    Jul 26, 2019 at 20:10
  • $\begingroup$ It could be useful to make a classification first. First, you classify data in larger values or lower values and them you construct a regression model for each one. In this way, you can be more accurate for larger values. $\endgroup$ Jul 27, 2019 at 9:53
  • $\begingroup$ @Skullgreymon thank you for your comment! It's really reasonable decision to consider a classification problem first. However, I'm interested to figure out especially, how it could be enhance in terms of regression. $\endgroup$
    – Lana
    Jul 27, 2019 at 10:04
  • $\begingroup$ Well, since you are using a Neural Network to do regression, I don't see how loss weight balancing would be a part of preprocessing? Did you try e.g. weighted batch sampling during for stochastic gradient descent? $\endgroup$ Jul 27, 2019 at 12:09

1 Answer 1


Neural Networks can in general be interpreted as a regression problem and as such, you could apply well known ways of dealing with this. This paper gives you a good introduction to different approaches. For instance you can upsample the minority class, or you could do loss weight balancing during training.

For instance, consider the data point $x_i$ that can belong to one of two classes $a$ and $b$. For instance, class $a$ is here the minority. Then you would, during training, multiply the calculated loss with the weight. $$L_i=\begin{cases}l_a\cdot x_i,&\text{if datapoint $x_i$ is in the minority class}\\l_b\cdot x_i,&\text{if datapoint $x_i$ is in the majority class}\end{cases}$$ with $l_a$ > $l_b$ which would be natural for this example.

You could also look into continous performance measures that could work nicely with imbalanced dataset. For instance the generalized dice (F1) score.

Referenced papers:

  • Provost 2000, Machine Learning from Imbalanced Data Sets 101
  • Sudre et. al 2017, Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations
  • $\begingroup$ @Lana did you not get it to work? $\endgroup$ Jul 28, 2019 at 8:48
  • 1
    $\begingroup$ Thank you for your answer, but does generalize dice loss work in regression task, if it evaluates multiple class segmentation loss (maybe I missunderstood something)? Although I don't see any mentions of regression task in the paper you have attached as an introduction, it's a useful material. $\endgroup$
    – Lana
    Jul 29, 2019 at 12:45
  • 1
    $\begingroup$ @Lana Well, I believe generalized dice only works for a two-class problem. Are you performing the regression differently from how you would train a normal supervised neural network (with soft classification)? If so, please specify how you are approaching your regression problem! If not, then most of the approaches described in the imbalanced-dataset paper, as well as weight balancing mentioned in the answer, should work just fine :) $\endgroup$ Jul 31, 2019 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.