2
$\begingroup$

I am currently writing a machine learning pipeline for my time series application. At the end of each month, I get the data gathered, normalize it ([0, 1]), retrain the ML model with the new observation only and predict future values.

Question

Should I be reading the entire dataset each time I get a new Observation, normalize the entire dataset, create the ML model, then predict?

How I got stuck:

  • Let's say I have 1 feature and at t-1 all of the values have min/max = [0, 1000]
  • At t, a new observation comes in with value = 1001
  • How should I normalize the new value given that the ML model has been trained with different min/max?

Thank you

$\endgroup$
1
  • $\begingroup$ If you know a rough maximum, I would try represent all numbers instead as binary. Lets say you are confident that numbers will never exceed 3000. You should then have 12 input nodes, which can either be 1 or 0, and then convert each number into binary. Lets say 1001, which would look like 001111101001. See how that works for you $\endgroup$ – Recessive Dec 13 '19 at 3:55
0
$\begingroup$

Really depends

Why? updating everything in production (pre-processing, fitting etc) can get extremely expensive. If you have some complex architecture it is not worth it.

Alternatives

  1. Approximate covariate shift if you know distribution of your future data you can adjust all your, for example normalisation parameters, in advance.

  2. Save your you future data every time you make prediction, it could be cheaper to quickly save your data in DB and depending on your system do updates weekly,monthly

$\endgroup$
1
  • $\begingroup$ Thank you for the answer Noah! Can you elaborate more on 2? I am already saving my prediction in DB. How can that be useful for my normalization? $\endgroup$ – Nick Dec 22 '19 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.