We are dealing with a small multilabel dataset (around 15k samples) of texts that is imbalanced. Some classes have more than 4k samples and others have around 700 samples. We are using a classifier based on fine-tuning BERT.
We decided to use several different binary classifiers at this moment for modeling our problem, instead of using a single multilabel classifier. So, each classifier is responsible for predicting if a sample is classified in some label or not.
Since our dataset is small, we have used the strategy of splitting the dataset into train/validation/test and we have built a single test set for all the classifiers. And the train set was developed for ensuring a balance between classes. We have used a single test set (that do not include instances used in train or validation of any classifier) in order to have the same test set for all the classifiers and for comparing their performance on the same basis.
Is it a good choice or it would be better to perform a cross-validation process? Or it does make sense to use first cross-validation and after this train/validation/test split that I have mentioned?
