I am developing a financial time-series prediction model using sklearn using
StandardScaler for scaling purposes. I train a model, and then use the model regularly on data as it comes in. The training must be done in batches due to the large data size. Right now, I am scaling each batch using a different scaler for training each batch, and for each test/real data batch.
My concern is that removing the mean from a subset of the data removes a good deal of information - the scaled data for an asset trading at at \$1500 is not clearly distinguishable from the data for a penny stock.
I'm wondering whether it is possible to - and whether I should - continually re-fit the same scaler used in training, so that data is scaled to remove the mean with respect to the entire dataset (ie. train + the actual data being evaluated), and whether this is desirable.