3
$\begingroup$

I'm new to data science but trying to get better

Here I have an attribute and plotting its histogram histogram

From what I know so far such a distribution is imbalanced and my goal is to equalize things a little bit right? Again from what I know, I have to transform this attribute to be more suitable for linear regression?

Is it obvious (to someone more experienced than me) which kind of transformation is applicable in this case?

Note that this is an attribute and not my target, this is not what I am trying to predict. This is one of the attributes to be used for predicting

$\endgroup$
2
  • 1
    $\begingroup$ You could try a logarithmic transformation. $\endgroup$
    – Emre
    Commented Oct 15, 2016 at 5:23
  • 1
    $\begingroup$ Because your data's distribution looks quasi exponential, and the logarithm is its inverse. Gerenuk explained the options well $\endgroup$
    – Emre
    Commented Oct 15, 2016 at 15:47

2 Answers 2

4
$\begingroup$

You have multiple options and you may choose the best by seeing the performance.

First one is to use a guess. For positive-only values the log-transform is a hot candidate to make sense (maybe correct for small values here to avoid exceedingly large negative transformed values).

Log-transform is natural if percentage increases have a particular real-world meaning. This is often the case for financial data. Why is log a hot candidate? You probably know that when there are addititve real world effects, the normal distribution often appears. Now, when you have multiplicative effects, you get the https://en.wikipedia.org/wiki/Log-normal_distribution .

Other common transformations are power transforms, where you take some power of the values. I don't think there are many more which are very common. Theoretically, your perfect transform would make the noise on the linear regression Gaussian, but no-one can tell what that would be and most likely reality isn't perfectly linear anyway.

A transform is more interesting when the transformed values follow a Gaussian distribution. But that is just a guess and in the end only final performance evaluation can tell more.

For a second option, be aware that you can force the transformed values to be any distribution you want. For example, if you take ranks of the values you get a uniform transformed distribution. You could even force it to be a Gaussian by a suitable mapping. However, in your case, this will lose the interesting bump on the right.

I think these are the most common options. In data science, nothing is ever obvious and most of the time you can only decide by performance evaluation (cross-validation with the whole model).

Conclusion:

Set up a performance test (cross-validation; not on final test set though if you like a fair, final evaluation) and try all of the following

  • Try untransformed. It might already be what has most information.
  • Try log-transform (while adding a small offset if you have some very small values)
  • Try power-transform if you like
$\endgroup$
0
$\begingroup$

I think what might help you given your problem is Synthetic Minority Over-Sampling Technique for Regression (SMOTER). There is some research on this topic. However, it remains less explored than its classification counterpart, as you have likely encountered.

I might suggest the paper cited below (and the conference it was presented at http://proceedings.mlr.press/v74/) depending on how interested you are in understanding it from a research perspective. I really appreciated the introduction of Gaussian noise in generating the synthetic observations.

If you're more interested in a practical solution, the first author has an R implementation available on her Github page. https://github.com/paobranco/SMOGN-LIDTA17

If Python is more of your persuasion, I recently released an entirely Pythonic implementation of the SMOGN algorithm that is now available and currently being unit tested. https://github.com/nickkunz/smogn

Branco, P., Torgo, L., Ribeiro, R. (2017). "SMOGN: A Pre-Processing Approach for Imbalanced Regression". Proceedings of Machine Learning Research, 74:36-50.http://proceedings.mlr.press/v74/branco17a/branco17a.pdf.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.