Suppose I have input values that represent the change of a stock share from each time step to the next. Now I want to feed these values into an LSTM Neural Net. My problem is that most values are between -0.000001 and 0.000001, but some values go up to 0.1 or -0.1. How can I effectively scale such a dataset. I've tried applying a log function (to the value +1, so e.g. log(1.000001)) and scaling between 0 and 1 and I've tried capping values at around 1.5. Do you have any suggestions how to deal with such extreme outliers better?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Play a bit with your data:
- try doing the scaling without these outliers and see what happens
- have you tried training the model regardless and see what happens? then train the model without the "outliers"
- you could also binarise the feature (and thus transform it into a category) - then in this case you don't have to deal with the large range while the information is still in there.
