How to proceed when training data change frequently (in production)?

Question

I'm working with a Recommendation System that would take as parameters a bunch of "tweets" a user see during his navegation on a mobile app. Every tweet has a property, like a category (which is a input from the Operational team). My recommedation at the end, is show the next "tweet" so it's interesting for the end user.

For the initial training and validation, everything work smoothly. I developed a NxM matrix and calculated the distance between each user. For the columns, I used the categories. As so, similar users read similar tweets of similar categories. I get the distance between these tweets and recommend the one which similar users liked most

My concern is that those categories change over time. New categories (aka dimensional space) expands. So supposed we had categories [Work | News | Health]. Next week, the Operational team would want to test new categories, like [Family, Fitness]. Not overwriting the olds, but launching new tweets with new categories that were not mapped on the training step

How can I maintain the model in prodution, since these new categories are always changing? The initital NxM matrix I built will not work, since the model in production was training using len(categories) that are less than actual. It's like saying model in production has a NxM matrix, but now tha data has NxM+2

Do I need to have a constante training step on the cloud? Since the model just take as a input the last few tweets a user saw, how can I keep the pace on new categories showing up?

Answer 1

The answer to your question really depends on buisness specifics.

If categories only update once a week, then probably the most reasonable solution is to retrain the model from scratch (as you suppose to do it regulary anyway, even with fixed number of categories).

If the updates are daily (or even more often), then you could start collecting an exhaustive list of categories (which is a good idea anyway), and possibly create a hierarchy of categories (e.g. #tennis is part of #sport, but not o/w). The latter will allow you to use some hierarchical loss (e.g. hierarchical cross-entropy), which in turn could help with cold-start (even if you get a completely new category, most likely it will be a part of some existing higher-level category). Again, even with exhaustive list of categories you do want to regularly retrain your model (how often is another question).

In general a completely new category means that you do not have any user-category pair in your training set, and this "item cold-start" problem do not have a "very good" solution. Alternatively, in your case it seems that the category itself is present, it is just not required at some of the weeks by a operational team. If this is the case, then I would follow the second option (get an exhaustive list and proceed..). Once you have an exhaustive list, if come of them are not required this week, simply skip it and move to the next most likely (for a particular user).