I have a small dataset(about 60 samples) and I need it to predict well for high target values. There are only a few high values and all models I tried perform poorly for these high values.
So I wonder what technics exist to make algorithm perform better for high values that can't be dismissed as outliers. You see, these few high values make MSE very large, because the models tends to underestimate these high values, predicting them to be 2 or more times smaller.
I have an idea to generate fake data for outliers, but I haven't found how to do it the right way for regression. Is it right to generate fake data for high values based on their proximity by mixing their features? So it would be smth like SMOTE, but instead of classes we have nearby values?
Or, perhaps, the other idea is to cluster targets by density and then to generate balanced clusters by generating fake data with SMOTE or ADASYN?
P.S. Note that I don't want to reduce effect of outliers. On the opposite, these high values are extremely important, so I want model to perform well for them.