Normalize new data and old model

I'm playing with machine learning and LSTM. My goal is to learn something new and work with real data. Currently, I'm trying to predict bitcoin price. I have understood the necessity to normalize data, but there is something that I don't get. When I use a scaler like MinMax (or any other scaler as I understand it), I give my train, validation and test sets to the scaler and I get a range of value easiest for tensorflow to work on.

After that, I can train my model and save it for later with model.save.

For this example, let's say that I trained my model on the 2021 (2021-01-01 to 2021-12-31) data for bitcoin price and the minimum price was \$20_000 and maximum was \$50_000.

Now it's 2022 and I got new data and I want to use my model saved last year for predicting the future! Let's say that minimum price here is \$15_000 and maximum \$55_000.

If I load my old model and add my new data (and normalize them) to it, the scaler need to be updated. How can I do that? I only find some basic tutorials with "finish dataset" or temperature prediction but not how to feed my model with new data.

I hope you understand my problem and that you will guide me to a solution. I can share my code if you want more information on my workflow.

Regards.

Assuming that you have univariate time series representing the evolution (in time) of the price value in a specific time-frame (e.g., 15 minutes, 4 hours, 1 day, etc.), a workaround for your problem would be to compute the first discrete derivative of such data before the training and the prediction phase. Formally, given a sequence $$n$$ of time-stamped observations:

$$S = \langle p_1,p_2,\ldots,p_n\rangle,$$

where $$p_i$$ represents the price value of the $$i$$-th time-frame (of Bitcoin, in your context), its corresponding first dicrete derivative sequence of $$n-1$$ observations is:

$$S' = \langle p_2 - p_1, p_3 - p_2, \ldots, p_n - p_{n-1}\rangle,$$

where $$p_{i+1} - p_i$$, for all $$1 \le i < n$$, represents the quantitative trend of the price action from the $$i$$-th time-frame to the $$(i+1)$$-th time-frame.

As an example, consider the following sequence and its corresponding first derivative sequence:

$$S_1 = \langle p_1, p_2, \ldots, p_n \rangle = \langle 10,20,40,60,50,50,20\rangle,$$

$$S_1' = \langle p_2 - p_1, p_3 - p_2, \ldots, p_n - p_{n-1}\rangle = \langle 10,20,20,-10,0,-30\rangle,$$

that could represent a $$7$$ days monitoring of the price action of Bitcoin in 2021. Observe that $$\min(S_1) = 10$$ and $$\max(S_1) = 60$$. Now, consider another sequence with its first derivative form:

$$S_2 = \langle p_1,p_2,\ldots,p_n \rangle = \langle 15,5,30,25,50,80,60 \rangle$$

$$S_2' = \langle p_2 - p_1, p_3 - p_2, \ldots, p_n - p_{n-1}\rangle = \langle -10,25,-5,25,30,-20 \rangle,$$

that could represent a (different) $$7$$ days monitoring of the price of Bitcon in 2022. Note that, in this case, $$\min(S_2) = 5$$ and $$\max(S_2) = 80$$, to follow your scenario.

One issue with such methodology is the loss of information when performing the transformation, but it immediately shows its strengths when facing problems like yours.

• Ah that's really smart, thank you very much. The data obtained will not be in range (0,1), this is not a problem? For your method to be effective, is there a minimum value number to add each time? Should the number of new values always the same? Feb 28, 2022 at 20:05
• The obtained data could be furtherly normalized to any interval, say, [x,y] (e.g., [0,1] using a min-max normalization), and this should not represent any issue. The last two questions are unclear. Could you formulate them better? Feb 28, 2022 at 21:24
• Ok, say I have one year of data, minute interval (so roughly 60*24*365). With your method, can I add 30 minutes, then 8 days (minute interval) and after that 5 minutes to my model? Or should I always take the same size interval like 30 minutes? Feb 28, 2022 at 22:48
• I have never tried the former, but, theoretically, you can use both. Nevertheless, I would suggest you keep the same granularity (i.e., interval, as you called it) between successive time-stamped price values; this makes perfect sense in your domain application. Feb 28, 2022 at 23:00
• Ok thank you very much I will try your suggestion. Have a nice day! Mar 1, 2022 at 0:21