So I have a
DataFrame which consists of order data from customer on an exchange.
I have a column
Dollars which is the dollar value of all trades conducted by customers in different currency pairs. I want to train a classifier on this data, so I'm normalizing the
Dollar column to get the values into the same range, but I think the normalized values seem quite.. odd. This is a snippet of my DataFrame:
Dollars | Normalized dollars 181447.50 | 9.10975e-06 281885.00 | 1.41523e-05 290786.00 | 1.45992e-05 70923.00 | 3.56076e-06 1121169.54| 5.62894e-05
These values seem tiny.
sklearn.preprocessing.normalize for this.
It's worth mentioning that the lowest
Dollars value is
0.06 while the largest is
5,605,847,772.52, which I assume is the explanation, but I was expecting a 0-1 range, but my largest normalized value is
0.157. Should I do more filtering on the dataset to remove extreme outliers as a general rule?
Applying the base 10 logarithm to dollars yields some more manageable results, though the normalized values are still fairly small.
ScaledDollars is the result of using
Dollars | Normalized dollars| Log10Dollars | LogNorm | ScaledDollars 181447.50 | 9.10975e-06 | 5.258751 | 0.00384193| -0.761916 281885.00 | 1.41523e-05 | 5.450072 | 0.00398171| -0.573336 290786.00 | 1.45992e-05 | 5.463573 | 0.00399157| -0.560028 70923.00 | 3.56076e-06 | 4.850787 | 0.00354388| -1.16404