1
$\begingroup$

I have a long list of continuous values like in the image below: enter image description here

The plot looks like this: enter image description here

How to handle such features? If I train the model with this, the model will not have the best precision, because there are a lot of outliners.

$\endgroup$

3 Answers 3

5
$\begingroup$

I suggest to try a log transformation. This has two potential benefits:

  1. The range of x values becomes smaller
  2. Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees)

Here are two toy examples to illustrate:

Toy example 1

s = np.random.lognormal(3, 1, 1000)
plt.hist(s, 100)
plt.show()

enter image description here

plt.hist(np.log(s), 100)
plt.show()

enter image description here

As you can see from the second plot the range of x values has become smaller and the transformed distribution resembles a normal distribution. Of course this is a highly artificial example since the non-transformed distribution is log_normal.

Toy example 2

s = np.random.geometric(0.2,100000)
plt.hist(s, 100)
plt.show()

enter image description here

plt.hist(np.log(s), 100)
plt.show()

enter image description here

This one does not look as nice (i.e. normally distributed) anymore as example 1 but still your range of x values has become more compact.

In case your data includes x values of $0$ you can use np.log(x+1). If your data included negative values it would become trickier (approaches for this case include signed log as described here or simply adding a constant value, like the minimum, to your data).

$\endgroup$
2
  • $\begingroup$ thank you very much! I will test it on my data. Is the range of x values becoming smaller better for the models instead of a long list of continuous values? $\endgroup$
    – nnikolay
    Commented Dec 31, 2019 at 15:02
  • 1
    $\begingroup$ @Mutatos it depends on the model. Some, like linear models and neural nets, are more sensitive to such transformations. Tree-based models, for example, are more robust. I'd apply a practical engineering-like approach: just try it out. $\endgroup$
    – Jonathan
    Commented Dec 31, 2019 at 16:16
4
$\begingroup$

Since you have a lot of values that are near 0, I would recommend to do a transformation very similar to a log transformation, but not a log transformation. It's called the bi-symmetric log transformation. I would propose that you read this.

$\endgroup$
3
$\begingroup$

As far as I can see, your values are positive, so you could simply take logs. You can also divide the variable by some value, say 10. You can make any type of linear transformation.

You could also see if scaling, e.g., min/max scaling would work for you. Preprocessing data | scikit-learn

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.