Multiclass classification task where each class is present only once in the test set

Question

I have a multiclass classification problem where, in the test set, there is only one entry for each possible class. In my particular problem we want to guess the author of a text, and we have 20 different authors. The test set contains 20 texts, one of each author.

I have many texts of each authors in the training data. I can't change the content of the test set, it's these specific texts that I need to classify.

• Does this kind of task has a name, so I can google previous work on it more easily ? I'm not talking about author identification specifically, but having 1 entry for each class.

• Do you have any suggestion of algorithm that would work well for this task ? Maybe something that gives ranked results or a confidence value ?

Thank you

Answer 1

With a single exemplar per class, there is a limit to appropriate evaluation metrics. There can be no estimate in the variability of performance.

It is best practice to look at test dataset only once so the current setup will have limited value.

A related machine learning field is one-shot learning.

It might be better to reframe the problem as a case study since there is not enough data for effective machine learning.

Answer 2

When talking about the test data, we can have any number of examples of each class for inference. It doesn't matter if it has 20 classes with one example each.

You can use a CNN classifier for this type of a problem but if one book can be written by more than one author you should use sigmoid activation in the last layer rather than using softmax.

You can see this article which address an usecase similar to yours - Link