# How to handle features with very broad range

I have a long list of continuous values like in the image below:

The plot looks like this:

How to handle such features? If I train the model with this, the model will not have the best precision, because there are a lot of outliners.

I suggest to try a log transformation. This has two potential benefits:

1. The range of x values becomes smaller
2. Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees)

Here are two toy examples to illustrate:

Toy example 1

s = np.random.lognormal(3, 1, 1000)
plt.hist(s, 100)
plt.show()


plt.hist(np.log(s), 100)
plt.show()


As you can see from the second plot the range of x values has become smaller and the transformed distribution resembles a normal distribution. Of course this is a highly artificial example since the non-transformed distribution is log_normal.

Toy example 2

s = np.random.geometric(0.2,100000)
plt.hist(s, 100)
plt.show()


plt.hist(np.log(s), 100)
plt.show()


This one does not look as nice (i.e. normally distributed) anymore as example 1 but still your range of x values has become more compact.

In case your data includes x values of $$0$$ you can use np.log(x+1). If your data included negative values it would become trickier (approaches for this case include signed log as described here or simply adding a constant value, like the minimum, to your data).

• thank you very much! I will test it on my data. Is the range of x values becoming smaller better for the models instead of a long list of continuous values? Commented Dec 31, 2019 at 15:02
• @Mutatos it depends on the model. Some, like linear models and neural nets, are more sensitive to such transformations. Tree-based models, for example, are more robust. I'd apply a practical engineering-like approach: just try it out. Commented Dec 31, 2019 at 16:16

Since you have a lot of values that are near 0, I would recommend to do a transformation very similar to a log transformation, but not a log transformation. It's called the bi-symmetric log transformation. I would propose that you read this.

As far as I can see, your values are positive, so you could simply take logs. You can also divide the variable by some value, say 10. You can make any type of linear transformation.

You could also see if scaling, e.g., min/max scaling would work for you. Preprocessing data | scikit-learn