Sustain learning separately - continuous learning

Question

This question is to seek suggestions on how to architect the continuous learning approach in distributed manner. Let me explain the situation:

1. In my classification problem, I have classes which can grow in large number over a period of time, as multiple customers from multiple domains tend to get attached to our multi-tenant application.
2. Many intents also need to be retrained many times as we need more real time data.
3. Many times, when a class is trained, it needs to be put in hibernate mode, as the customers are seasonal. They may get awake after few months and so can not keep their training data every time in the training dataset. It leads to slow training as well as imbalanced dataset.

What I want is:

1. I want that if I train a class today, and I know that it is not going to be called for next few months, I just want to retain its learnings somewhere.
2. Rather than keeping its labeled dataset with me and I retrain again in future when need arises, I want that just the learning (be it weights or the model itself), which can be just plugged back in to running environment.

Suggestions needed:

1. How to retain the learnings separately on disk? Should it be a separate model? Drawback of a separate model is that if I have to bring up multiple models to life, that will consume a lot of runtime memory. Also, how to architecture would be of a model having a single class??
2. If want to keep just weights stored somewhere, then how to fit them back on a model which has learnt more data, while those weights were sleeping on disk?
3. If I take approach of transfer learning, then what base model would be trained on? new classes may not be known during the old time of building base model.

Answer 1

One option is incremental machine learning (sometimes called online machine learning). In incremental machine learning, existing model weights are updated with new data as the new data is available for training.

If the size of the models is a concern, there is work on sparse models. One example is the Pathway architecture where a single, very large model only activates a small subset of pathways through the network as needed for a specific decision. The model itself learns which parts of the network are needed for which predictions. Pathway architectures can scale to 100s of billions of parameters.