What to do if my dataset have only One instance for class in classification?

Question

I am working on a benchmark dataset for text classification. The dataset has about 300 classes, and approximately 50 of these classes have a single instance. In a paper that used fine-tuning BERT, the authors split the dataset into training, validation, and test sets. However, they did not increase the number of instances through oversampling or augmentation. This brings up the topic of what happens to classes with a single instance. In which split should they be included?

Answer 1

Train, Validations, and Test Split Sets cannot give unreliable evaluations of the model's performance. Because it will not be able to adapt well to the new instances of those classes.

Stratified sampling may not be effective in this case, Because of the low Instances. In that case, The potential solution is K-Fold. It divides the dataset into k numbered folds. Each contains the same portion of the dataset. Every fold is trained and evaluated, and the results are averaged. Which gives the perfect prediction.

Here is the code example[1]:

# scikit-learn k-fold cross-validation
from numpy import array
from sklearn.model_selection import KFold
# data sample
data = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6])
# prepare cross-validation
kfold = KFold(3, True, 1)
# enumerate splits
for train, test in kfold.split(data):
    print('train: %s, test: %s' % (data[train], data[test]))

Reference:

1. https://machinelearningmastery.com/k-fold-cross-validation/