How to handle "unknown" category in machine learning classification problems?

Question

Tutorial problems come in the form of binary or mult-class classification where data are all properly labelled. In real-life applications, there are incoming data that do not belong to any category and cannot be classified. How can we handle these data which fall into "unknown" category? The universe of "unknown" can be far more than "known". So, the data for "unknown" can be too much and lead to class imbalance. How do we train the model to deal with "unknown" data? Or do we ignore it?

Answer 1

I think this is one of those topics with the most frustrating answer - it depends.

To your questions:

How can we handle these data which fall into "unknown" category?

There are many ways of doing this. Some are very simple and some are more complex, but they all depend on you understanding your data and what exactly causes the missingness - e.g. is the data missing at random or is there a specific cause driving it?

Some techniques to treat missing values (in increasing order of complexity):

1. Exclude all missing values. This may be fine if you have a large amount of data and few missing values (not always the case - you allude to this by mentioning the remaining data set may be imbalanced).
2. Replace / group the missing values with an appropriate value - e.g. replace missing values with the mean of the variable / group missings with the most populous level.
3. Impute missing values using models / equations - e.g. Multivariate Imputation by Chained Equations (MICE).

How do we train the model to deal with "unknown" data?

This depends on the model or technique you are using. Some techniques deal with missing values well (e.g. xgboost), while others do not (e.g. R's ranger implementation of random forests). You should take into account the model you are using when deciding on how to treat your missing values.

Or do we ignore it?

Ignore missing values at your peril!

Hope that helps!

Answer 2

I suggest these methods:

1. do oversample or undersample

2. instead of accuracy consider AUC or F1-score.

3. define the cost of the different class of prediction vs actuality.

4. try to find some rules to find 'unknown' then remove them from training data.

good luck.

Answer 3

It depends on what kind of machine learning you are working with. Supervised machine learning models require labeled data in order to distinguish between various categories and detect patterns. They are often more popular due to the fact that they are often more successful than their unsupervised counterparts when labeled data is plentiful.
Unsupervised models, on the other hand, require no labeled data at all. They just survey the data and determine similarities, which can be advantageous in certain circumstances where the amount of unlabelled data is large enough to give the model the resources it needs to learn effectively. Examples of these would be deep-belief networks, cluster analysis, and autoencoders.
Semi-supervised models are a bit of a combination of the two and can learn from a mixture of labeled and unlabeled data. The labeled data gives the network a basis on which to draw verifiable conclusions.
Hope this helps.