0
$\begingroup$

I am developing a financial time-series prediction model using sklearn using StandardScaler for scaling purposes. I train a model, and then use the model regularly on data as it comes in. The training must be done in batches due to the large data size. Right now, I am scaling each batch using a different scaler for training each batch, and for each test/real data batch.

My concern is that removing the mean from a subset of the data removes a good deal of information - the scaled data for an asset trading at at \$1500 is not clearly distinguishable from the data for a penny stock.

I'm wondering whether it is possible to - and whether I should - continually re-fit the same scaler used in training, so that data is scaled to remove the mean with respect to the entire dataset (ie. train + the actual data being evaluated), and whether this is desirable.

$\endgroup$

1 Answer 1

1
$\begingroup$

I was able to find some consensus in various stack* sites - the answer is apparently to scale test data with respect to the train data - ie. (testData - mean(trainData)) / sd(trainData), or with sklearn scaler.fit(train_data); scaler.transform(test_data).

https://stats.stackexchange.com/questions/174823/how-to-apply-standardization-normalization-to-train-and-testset-if-prediction-i

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.