I'm creating a multilabel classification approach based on sentence embeddings applied to text taken from a chatbot. We have the following:
- a training dataset of 2,500 lines, where each line is a sentence associated with a particular label (the same sentence can end up with several labels)
- a production dataset made up of sentences taken from our previous chatbot version. Each sentence has been carefully annotated with the corresponding labels. (The number of labels per sentence ranges from 0 to 5 in most cases). There are about 1000 annotated sentences in this dataset.
I'd like to maximize the performance and generalizability of the approach, but I have the following limitations:
- the production dataset doesn't cover all possible labels, as some labels are rare but still important to detect
- the training dataset covers all labels, but has been created from imaginary sentences.
The current process is as follows:
- Evaluate performance with MultilabelStratifiedKFold on the training dataset
- Select the best model and train the final model with all data
- Evaluate performance on the production dataset and choose a threshold to separate what the model considers relevant from the rest (for the moment, I'm choosing the point that balances micro averaged precision and micro averaged recall on the production dataset).
My question is this: When you choose the threshold, how do you avoid overfitting as much as possible in this context? We don't want overfitting on either the training or production datasets. Wouldn't optimizing the threshold on the production dataset bias the approach towards the labels available in the production dataset? In the same way, optimizing the threshold on the training dataset might bias the approach towards less "realistic" phrases from the real production context.
What do you think?