# Poor performance of regression model for imbalanced data

I am trying to train a neural network model to solve a regression problem. The specificity of my dataset is that it has something like an exponential distribution of target values (imbalanced). Therefor, the model seems to output just values less than 2 (if the range is [0,6]), for instance, and it absolutely ignores bigger target's values, which have a smaller performance in the dataset. How is it possible to improve the model's results in such a case?

For example, when it comes to a multiclass classification, we can weigh penalties for errors on a smaller class to enhance performance with imbalanced data. Are there any tricks in terms of regression? Which loss-functions could be useful? It seems, that MSE loss function is more preferable than RMSE. Is it more powerful loss functions for this problem?

There is a paper about such a problem of imbalanced regression (http://proceedings.mlr.press/v74/branco17a/branco17a.pdf) which might be helpful to someone. However, I'm more interested in special tricks for a neural network, not pre-processing approaches (I can't generate more data for example).

• Hi, Lana! Could you be more specific in regards to how your dataset is structured? The specifications of this exponential distribution could be useful, or at all what specifically you are trying to achieve? Is it exponential regression? I'm also confused how this is multiclass? – Andreas Storvik Strauman Jul 26 '19 at 16:02
• @AndreasStorvikStrauman I'm trying to enhance performance on instances, which have a smaller representation in dataset, because now the model ignores such cases. The references to a multiclass classification is just an example, how a similar problem could be solved not for regression. It's not an exponential regression. – Lana Jul 26 '19 at 20:10
• It could be useful to make a classification first. First, you classify data in larger values or lower values and them you construct a regression model for each one. In this way, you can be more accurate for larger values. – Skullgreymon Jul 27 '19 at 9:53
• @Skullgreymon thank you for your comment! It's really reasonable decision to consider a classification problem first. However, I'm interested to figure out especially, how it could be enhance in terms of regression. – Lana Jul 27 '19 at 10:04
• Well, since you are using a Neural Network to do regression, I don't see how loss weight balancing would be a part of preprocessing? Did you try e.g. weighted batch sampling during for stochastic gradient descent? – Andreas Storvik Strauman Jul 27 '19 at 12:09

For instance, consider the data point $$x_i$$ that can belong to one of two classes $$a$$ and $$b$$. For instance, class $$a$$ is here the minority. Then you would, during training, multiply the calculated loss with the weight. $$L_i=\begin{cases}l_a\cdot x_i,&\text{if datapoint x_i is in the minority class}\\l_b\cdot x_i,&\text{if datapoint x_i is in the majority class}\end{cases}$$ with $$l_a$$ > $$l_b$$ which would be natural for this example.