Normalize new data and old model

Question

I'm playing with machine learning and LSTM. My goal is to learn something new and work with real data. Currently, I'm trying to predict bitcoin price. I have understood the necessity to normalize data, but there is something that I don't get. When I use a scaler like MinMax (or any other scaler as I understand it), I give my train, validation and test sets to the scaler and I get a range of value easiest for tensorflow to work on.

After that, I can train my model and save it for later with model.save.

For this example, let's say that I trained my model on the 2021 (2021-01-01 to 2021-12-31) data for bitcoin price and the minimum price was \$20_000 and maximum was \$50_000.

Now it's 2022 and I got new data and I want to use my model saved last year for predicting the future! Let's say that minimum price here is \$15_000 and maximum \$55_000.

If I load my old model and add my new data (and normalize them) to it, the scaler need to be updated. How can I do that? I only find some basic tutorials with "finish dataset" or temperature prediction but not how to feed my model with new data.

I hope you understand my problem and that you will guide me to a solution. I can share my code if you want more information on my workflow.

Regards.

Answer 1

Assuming that you have univariate time series representing the evolution (in time) of the price value in a specific time-frame (e.g., 15 minutes, 4 hours, 1 day, etc.), a workaround for your problem would be to compute the first discrete derivative of such data before the training and the prediction phase. Formally, given a sequence $n$ of time-stamped observations:

$$S = \langle p_1,p_2,\ldots,p_n\rangle,$$

where $p_i$ represents the price value of the $i$-th time-frame (of Bitcoin, in your context), its corresponding first dicrete derivative sequence of $n-1$ observations is:

$$S' = \langle p_2 - p_1, p_3 - p_2, \ldots, p_n - p_{n-1}\rangle,$$

where $p_{i+1} - p_i$, for all $1 \le i < n$, represents the quantitative trend of the price action from the $i$-th time-frame to the $(i+1)$-th time-frame.

As an example, consider the following sequence and its corresponding first derivative sequence:

$$S_1 = \langle p_1, p_2, \ldots, p_n \rangle = \langle 10,20,40,60,50,50,20\rangle,$$

$$S_1' = \langle p_2 - p_1, p_3 - p_2, \ldots, p_n - p_{n-1}\rangle = \langle 10,20,20,-10,0,-30\rangle,$$

that could represent a $7$ days monitoring of the price action of Bitcoin in 2021. Observe that $\min(S_1) = 10$ and $\max(S_1) = 60$. Now, consider another sequence with its first derivative form:

$$S_2 = \langle p_1,p_2,\ldots,p_n \rangle = \langle 15,5,30,25,50,80,60 \rangle$$

$$S_2' = \langle p_2 - p_1, p_3 - p_2, \ldots, p_n - p_{n-1}\rangle = \langle -10,25,-5,25,30,-20 \rangle,$$

that could represent a (different) $7$ days monitoring of the price of Bitcon in 2022. Note that, in this case, $\min(S_2) = 5$ and $\max(S_2) = 80$, to follow your scenario.

One issue with such methodology is the loss of information when performing the transformation, but it immediately shows its strengths when facing problems like yours.