How to continue incremental learning when a categorical variable has been assigned additional category labels?

Question

Please help answer this question or point me to any resource.

There is a model in an environment where training happens with new data and the data is discarded after training is completed. This keeps on happening in cycles. Hence we dealing with models allowing "partial fit"

In this model, there is a training variable X , which is a categorical variable. X has some previously known categories C and its encoded as "one hot vector" of length D .

Now suppose we are now observing a new category of X, so we need to extend encoding of C to length D+1 .

The question is, how to do this without losing all the previously learned knowledge by model.

Training again is not an option since we have discarded all the previously held data and only new data can be used for training.

There is already a question Updating One-Hot Encoding to account for new categories However the answer is either to use entire data to create the "one hot encoding" (which is not possible in this environment since new data is in future) , or to omit the usage entirely (thus loosing some information).

Answer 1

Looks like a special case for incremental learning / life-long learning. There are many papers in this line of work:

https://arxiv.org/abs/1611.07725

https://openreview.net/forum?id=BkloRs0qK7

Most try to preserve the model parameters in a way that reduces the loss of previous knowledge ("catastrophic forgetting"), but some also try to find (or keep) some samples and preserve the original model's response on those samples.

Answer 2

Consider entity embeddings which use "semantics" / "intrinsic properties".

"Entity Embeddings of Categorical Variables" https://arxiv.org/abs/1604.06737

Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables.

If this looks good then please advise and will share more resources.