So I have a DataFrame
which consists of order data from customer on an exchange.
I have a column Dollars
which is the dollar value of all trades conducted by customers in different currency pairs. I want to train a classifier on this data, so I'm normalizing the Dollar
column to get the values into the same range, but I think the normalized values seem quite.. odd. This is a snippet of my DataFrame:
Dollars | Normalized dollars
181447.50 | 9.10975e-06
281885.00 | 1.41523e-05
290786.00 | 1.45992e-05
70923.00 | 3.56076e-06
1121169.54| 5.62894e-05
These values seem tiny.
I've used sklearn.preprocessing.normalize
for this.
It's worth mentioning that the lowest Dollars
value is 0.06
while the largest is 5,605,847,772.52
, which I assume is the explanation, but I was expecting a 0-1 range, but my largest normalized value is 0.157
. Should I do more filtering on the dataset to remove extreme outliers as a general rule?
UPDATE:
Applying the base 10 logarithm to dollars yields some more manageable results, though the normalized values are still fairly small. ScaledDollars
is the result of using sklearn.StandardScaler
on Log10Dollars
:
Dollars | Normalized dollars| Log10Dollars | LogNorm | ScaledDollars
181447.50 | 9.10975e-06 | 5.258751 | 0.00384193| -0.761916
281885.00 | 1.41523e-05 | 5.450072 | 0.00398171| -0.573336
290786.00 | 1.45992e-05 | 5.463573 | 0.00399157| -0.560028
70923.00 | 3.56076e-06 | 4.850787 | 0.00354388| -1.16404