I am working on a benchmark dataset for text classification. The dataset has about 300 classes, and approximately 50 of these classes have a single instance. In a paper that used fine-tuning BERT, the authors split the dataset into training, validation, and test sets. However, they did not increase the number of instances through oversampling or augmentation. This brings up the topic of what happens to classes with a single instance. In which split should they be included?
Train, Validations, and Test Split Sets cannot give unreliable evaluations of the model's performance. Because it will not be able to adapt well to the new instances of those classes.
Stratified sampling may not be effective in this case, Because of the low Instances. In that case, The potential solution is K-Fold. It divides the dataset into k numbered folds. Each contains the same portion of the dataset. Every fold is trained and evaluated, and the results are averaged. Which gives the perfect prediction.
Here is the code example:
# scikit-learn k-fold cross-validation from numpy import array from sklearn.model_selection import KFold # data sample data = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]) # prepare cross-validation kfold = KFold(3, True, 1) # enumerate splits for train, test in kfold.split(data): print('train: %s, test: %s' % (data[train], data[test]))