5
$\begingroup$

Please help answer this question or point me to any resource.

There is a model in an environment where training happens with new data and the data is discarded after training is completed. This keeps on happening in cycles. Hence we dealing with models allowing "partial fit"

In this model, there is a training variable X , which is a categorical variable. X has some previously known categories C and its encoded as "one hot vector" of length D .

Now suppose we are now observing a new category of X, so we need to extend encoding of C to length D+1 .

The question is, how to do this without losing all the previously learned knowledge by model.

Training again is not an option since we have discarded all the previously held data and only new data can be used for training.

There is already a question Updating One-Hot Encoding to account for new categories However the answer is either to use entire data to create the "one hot encoding" (which is not possible in this environment since new data is in future) , or to omit the usage entirely (thus loosing some information).

$\endgroup$
  • 1
    $\begingroup$ Either set aside a reasonable number of bits for expansion and use feature hashing, or learn an embedding for your categorical variables and keep the width fixed. I'll let others elaborate. Welcome to the site! $\endgroup$ – Emre Mar 19 '18 at 7:11
  • $\begingroup$ did you find a solution to this? $\endgroup$ – swalkner Apr 2 at 11:23
3
$\begingroup$

Looks like a special case for incremental learning / life-long learning. There are many papers in this line of work:

https://arxiv.org/abs/1611.07725

https://openreview.net/forum?id=BkloRs0qK7

Most try to preserve the model parameters in a way that reduces the loss of previous knowledge ("catastrophic forgetting"), but some also try to find (or keep) some samples and preserve the original model's response on those samples.

$\endgroup$
1
$\begingroup$

Consider entity embeddings which use "semantics" / "intrinsic properties".

"Entity Embeddings of Categorical Variables" https://arxiv.org/abs/1604.06737

Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables.

If this looks good then please advise and will share more resources.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.