2
$\begingroup$

During prediction phase, fully trained supervised models may have to deal with data representing new classes, that weren't part of the training and test sets. A real world example for this issue is Supervised Condition Monitoring, where a classifier can be trained on data representing several conditions of a mechanical system. Since it is impossible to know and/or generate data for all possible conditions, such a classifier may encounter data representing new and unknown classes during runtime.

What I am looking for, is the general term for this issue and approaches that address it. Semi-Supervised-Learning doesn't seem to be a good fit here, since in my example, all training and test data is labeled, and new classes only occur during runtime. For similar reasons, Data Drift doesn't apply, since the drift only refers to data of known classes. One-Class-Classification would not help either, since the example clearly represents a multiclass scenario.

A good approach to address the issue would be classifiers, that for each test sample and each known class output the probability of the test sample belonging to this class. For a test sample belonging to an unknown class, all these probabilities should be low.

Does anyone know the scientific term for this general issue and can point me into the direction of relevant research and possible solutions?

$\endgroup$

2 Answers 2

2
$\begingroup$

One option would be to take a Bayesian approach by estimating the confidence in the class prediction. If all of the confidences are below a threshold, predict "new class".

Another option is zero-shot learning (ZSL) which goes one step further and attempts to predict a specific class for the unobserved classes. ZSL often requires auxiliary information, which includes distinguishing properties of the items.

$\endgroup$
1
$\begingroup$

It might not be exactly what you're asking, but your example describes a clear use case for one-class classification: there is a representative sample only for the 'good' class, anything else is is a fault condition. It's different from regular classification because it does not rely on discriminating between the classes: in regular classification (usually with one-against-all), the model predicts the most likely class with respect to the other classes. As a consequence, the model is not meant to deal with cases which are completely different from the training set in the regular classification setting. By contrast, in the one-class classification setting the model attempts to represent the class on its own, so in theory is best prepared to deal with any completely different case which doesn't belong to the class.

$\endgroup$
2
  • $\begingroup$ One -class -classification doesn't exactly cover the issue. What I am looking for is a multiclass classifier capable of "predicting" data representing "unknown classes". Nor is this a semi-supervised scenario, since in my example, training/ test data would be completely labeled. $\endgroup$ Jan 8, 2023 at 20:27
  • $\begingroup$ @user1934212: to my knowledge there's no such thing, because supervised classification relies on the assumption that the distribution of the test set is the same (or is similar) as the training set. The set of classes in particular must be known, and there must be a representative sample of each class in the training set. I guess you could consider as scenario where you would first use a non-parametric clustering approach to separate the classes, and then maybe a distance-based approach to map some of the clusters with known classes. $\endgroup$
    – Erwan
    Jan 8, 2023 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.