1
$\begingroup$

So I have a DataFrame which consists of order data from customer on an exchange.

I have a column Dollars which is the dollar value of all trades conducted by customers in different currency pairs. I want to train a classifier on this data, so I'm normalizing the Dollar column to get the values into the same range, but I think the normalized values seem quite.. odd. This is a snippet of my DataFrame:

Dollars   | Normalized dollars
181447.50 | 9.10975e-06
281885.00 | 1.41523e-05
290786.00 | 1.45992e-05
70923.00  | 3.56076e-06
1121169.54| 5.62894e-05

These values seem tiny.

I've used sklearn.preprocessing.normalize for this.

It's worth mentioning that the lowest Dollars value is 0.06 while the largest is 5,605,847,772.52, which I assume is the explanation, but I was expecting a 0-1 range, but my largest normalized value is 0.157. Should I do more filtering on the dataset to remove extreme outliers as a general rule?

UPDATE: Applying the base 10 logarithm to dollars yields some more manageable results, though the normalized values are still fairly small. ScaledDollars is the result of using sklearn.StandardScaler on Log10Dollars:

Dollars   | Normalized dollars| Log10Dollars | LogNorm   | ScaledDollars
181447.50 | 9.10975e-06       | 5.258751     | 0.00384193| -0.761916
281885.00 | 1.41523e-05       | 5.450072     | 0.00398171| -0.573336
290786.00 | 1.45992e-05       | 5.463573     | 0.00399157| -0.560028
70923.00  | 3.56076e-06       | 4.850787     | 0.00354388| -1.16404
Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

You should use scaling instead of normalizing. By normalizing the column you make the norm 1, while you are interested in making the expected value 0 with a standard deviation of 1. With the norm, the more samples you have the smaller your samples will become. Use the scikit-learn StandardScaler instead. Despite this, with these huge numbers you will still get weird results (although most of them will be negative with this skew). What could be an alternative is using a log transformation on your dollars first, and then scaling them. Conceptually you usually look at relative values between dollars, the difference between 10 and 20 dollars is similar to that of 10,000 and 20,000 which the log transformation will capture.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the input. I've tried doing some logarithmic transformation on the dollar values and have tried StandardScaler. I have to read up on it though to make much sense of the results. $\endgroup$
    – Khaine775
    Oct 24, 2017 at 13:13
2
$\begingroup$

It is not surprising to obtain values that tiny because all your values are being scaled down by $5,605,847,772.46$ after being subtracted by $0.06$.

How about doing lognormalization? Apparently it is not uncommon to use this on financial data like explained here.

EDIT (For completeness of answer)

After taking the Logarithm of each of your value, you might want to use MinMaxScaler() instead of L2 Norm of scikit-learn if you want your values to lie beween 0-1.

If an element '$x_i$' in Log10Dollars after Normalization/Scaling is represented as '$\bar{x_i}$' then,

  • Using L2 Norm from sklearn: $\bar{x_i} = \frac{x_i}{\sqrt{\sum_{i=1}^{n}{x_i^2}}}$
  • Using MinMaxScaler(): $\bar{x_i} = \frac{x_i−min(x)}{max(x)−min(x)}$

The second representation will give you $\bar{x_i} \in [0,1]$.

Improve this answer
$\endgroup$
4
  • $\begingroup$ That's a good idea. I've tried implementing that. $\endgroup$
    – Khaine775
    Oct 24, 2017 at 13:02
  • $\begingroup$ If you want the data to lie between 0-1 range, you could use MinMaxScaler() after using taking the logarithm instead of using L2 Norm of sklearn. $\endgroup$
    – tomar__
    Oct 24, 2017 at 13:23
  • $\begingroup$ I could do that, but I'm not entirely sure I can explain the differences between applying MinMaxScaler() and the L2 Norm and the practical consequences. $\endgroup$
    – Khaine775
    Oct 24, 2017 at 13:26
  • 1
    $\begingroup$ Using L2 Normalization, each value $x_i$ in your Log10Dollars column is scaled by $\sqrt{(\sum_{i=1}^n(x_i^2)}$. This is not the same as doing $\frac{x_i - min(x)}{max(x)-min(x)}$ which $MinMaxScaler()$ will perform and give you your values in 0-1 range. $\endgroup$
    – tomar__
    Oct 24, 2017 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.