Recently i was working on a problem where my target variable is a continuous variable, with highly right skewed data. Please refer to image below

enter image description here

If i want to create regression this please suggest the best approach :

    1. Should i do any transformation ?
    2. It has very long tail should i ignore them or how to tackle them?

    3. Any modifications required in the loss function (Using SSE Right Now)?
    4. What is the best evaluation metrics (Using MAPE right now)?

Till Now i have removed outliers after 99.5 percentile values and with a XGBoost model i have been able to reach to 70% MAPE.

  • $\begingroup$ For what reason do desire normality? $\endgroup$
    – Dave
    Jan 10 at 0:49

You might consider a log transform or a square root transform to reduce the skew but they work on positive values only. This might also help you with the long tail of values by "drawing them in closer".

  • $\begingroup$ I thought about log transform but then if you look at my data most values lie between 0.01 to 6, i am worried about their differentiation after log transformation $\endgroup$ Jan 9 at 15:45
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jan 9 at 16:20
  • $\begingroup$ The log transform exaggerates differences in values between 0 and 1 and reduces differences in values between 1 and infinity. So the values between 0.01 and 1 should be easily distinguishable after the log transform. I don't think you'll run into issues with values between 1 and 6 either, but it depends on your specific case of course. $\endgroup$ Jan 9 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.