I have a very large dataset with timestamp data. Till now, I loaded the whole dataset in order to train some models using python (statsmodels.api, statsmodels.formula.api, keras.models.Sequential). Now, I would like to train models with all historical data just one time, and then update the model just with new data (each day I have new samples). Is it possible and how with these python libraries? Thanks!
This is what Machine Learning models are used for..(to predict what they think will happen in the near time based on the data they have been fed into....)
A simple answer is this
The first couple days of data are the most important , you need to gauge and have a watch over your model..
As with any incremental learning, you can learn more recent stuff but also underfit the past significantly more than without learning the new stuff. Therefore, it's why there's a monitoring to perform on models which use incremental learning (especially on production systems).
Provided that other things remains similar with time..(same Preprocessing, same scaling etc..)
But let's say if you have tons of data everyday, then nearly at the month end, the model can be completely trained on the last months data or else checkpoints needs to be kept
"train models with all historical data just one time, and then update the model just with new data (each day I have new samples"
Seems you are referring to incremental ML algorithm - beyond "experimental data science" mindset.
See "Incremental principal component analysis (IPCA)" at https://scikit-learn.org/stable/auto_examples/decomposition/plot_incremental_pca.html for an example of algorithm design and use.
If that helps then you might want to look into
- "Concept Drift: Monitoring Model Quality in Streaming Machine Learning Applications" https://www.lightbend.com/blog/concept-drift-monitoring-model-quality-in-streaming-ml-applications
- "Incremental Principal Component Analysis Exact implementation and continuity corrections" https://arxiv.org/abs/1901.07922