How to test/train a model for realtime data with new data points and classes in a ML pipeline

Question

First, For a text classification problem, if I have trained the model on 2 classes and it gives good accuracy. Now, when I use the model in real-time, there is a completely new class from a totally different business team which I need to use the trained model and test this data on. Here, obviously the model will have worse results, and also the pre-processing or any other data cleaning could be way different. So, the question is - Is the only possible way is to re-train the model again?

Second, In a segmentation problem, the approach is to use K-Means clustering and group customers. On a live environment, how do we handle new data points? How do we know which group does the data point belong to? Coz the model has been standardized/normalized properly and then fitted. The same cannot be expected for a new data point. How do we handle this?

Answer 1

In both cases you need to know that the already trained model is not written on the stone. Specially the second question "needs" change of model.

To be more detailed, you are talking about Online Learning in general. Many ML algorithms have online versions. Here, for example, you see the online version of K-means for second question. But in general, online learning is nothing more than adapting a model to the situation in which data comes one sample after the other (like your question). So for basic usage, you can simply adapt your model respectively in a regular manner. this can not be fully online but with a time gap you can retrain the model with new data points which are not assigned to any cluster yet. For fully online-algorithms you can just refer to the link above and start researching.

For the first question I have a meta-question. How do you know that you have a totally new class of texts? So first you need to detect the change and then react correspondingly. To detect the presence of a new class, I propose looking at the logit of the ML algorithm. For instance, if it is a probabilistic approach, you may look at probability of assignment to any present class and say if all less than a threshold and the number of out-of-vocabulary words of new document according to current vocabulary is above a threshold, I create a new class. Not state-of-the-art but for basic usages it will work I assume. Not much to do with preprocessing in a very online manner if it needs a big change. Preprocessing should either be simple and general or very specific according to the class. The first does not need a change and the second needs a bunch of samples from new class to be decided. leave it simple and general for all (like removing strange punctuations, lower-casing, etc.)