I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values.

So I wonder what technics exist to make algorithm perform better for high values that can't be dismissed as outliers. You see, these few high values make MSE very large, because the models tends to underestimate these high values, predicting them to be 2 or more times smaller.

I have an idea to generate fake data for outliers, but I haven't found how to do it the right way for regression. Is it right to generate fake data for high values based on their proximity by mixing their features? So it would be smth like SMOTE, but instead of classes we have nearby values?

Or, perhaps, the other idea is to cluster targets by density and then to generate balanced clusters by generating fake data with SMOTE or ADASYN?

P.S. Note that I don't want to reduce effect of outliers. On the opposite, these high values are extremely important, so I want model to perform well for them.

  • $\begingroup$ You should post a plot of the model (residuals, etc). Your variable may be non-linear or may require some transformation / weighting. $\endgroup$ – user2974951 Oct 5 '18 at 12:42

I think what might help you given your problem is Synthetic Minority Over-Sampling Technique for Regression (SMOTER). It will generate synthetic observations for the regions of interest in your response variable. In your case, high values.

I might suggest the paper cited below for context. If you're more interested in a practical solution, the first author has an R implementation available on her Github page. https://github.com/paobranco/SMOGN-LIDTA17

If you're a Python user, I recently open-sourced an entirely Pythonic implementation of the SMOGN algorithm that is now available on PyPI and Github. https://github.com/nickkunz/smogn

I hope this helped!

Branco, P., Torgo, L., Ribeiro, R. (2017). "SMOGN: A Pre-Processing Approach for Imbalanced Regression". Proceedings of Machine Learning Research, 74:36-50.http://proceedings.mlr.press/v74/branco17a/branco17a.pdf.

  • 1
    $\begingroup$ Thank you! I looked for that implementation in Python a year ago, but it wasn't available back then, only in R. Great it's in Python now! $\endgroup$ – DmytroSytro Dec 17 '19 at 10:00
  • $\begingroup$ You bet! I wish I would've stumbled upon the paper sooner. The package is still early in development, but I will continue to make improvements. If you have any suggestions, please let me know. Thanks! $\endgroup$ – Nick Kunz Dec 18 '19 at 2:32

When it comes to outliers, one possible way of reducing the effects of the same is by means of a weighting mechanism.

When a standard regression is run on an entire dataset, all the observations in the dataset are being weighted equally.

Therefore, you could choose to run a robust regression where you use a weighting system, e.g. Huber, Bisquare, to adjust the weights so as to place a smaller weight on outliers in your dataset.

One way of detecting outliers in the first instance can be through the use of Cook's Distance, whereby if the distance is greater than 1 (as a rule of thumb), then the observation could be deemed to be an outlier.

You might find the following references helpful:

  • $\begingroup$ But that's the opposite of what I want. I want the model to perform better for outliers. If I reduce weights for them, it'll perform worse. $\endgroup$ – DmytroSytro Oct 8 '18 at 7:53
  • 1
    $\begingroup$ Apologies, I misunderstood your question. That said, the principle is the same. If you wanted outliers to feature more prominently in the results of the model, then you need to adjust the weights so as the outliers have a higher weighting. I would recommend checking out the rlm package in R, as you might find that this gives you more flexibility in adjusting the weights as discussed: stat.ethz.ch/R-manual/R-patched/library/MASS/html/rlm.html $\endgroup$ – Michael Grogan Oct 11 '18 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.