Fine-tune zero-shot classification model multi-label

Question

I started a small project where I am trying to fine-tune a zero-shot classification model on a proprietary dataset. I was thinking to use the NLI approach, building contradiction and entailment statements for each of my sentences/labels pairs.

I have a dataset with sentences and for each of them multiple true labels.

However, I am not sure on what is the best way to approach this, given that in literature I have only seen the case where there is only one label per sentence.

Making one example:

Sentence 1. Classes = ['A','B','C']

Should I build my dataset generating three different samples

Sentence 1. This is about 'A' + Entailment label Sentence 1. This is about 'B' + Entailment label Sentence 1. This is about 'C' + Entailment label

or generating only one as follows:

Sentence 1. This is about A, B, C. + Entailment label

I am happy to hear any other ideas on this.

Thanks a lot!

Answer 1

You can use the NLI approach. The NLI task requires the model to identify the relationship between two sentences: a hypothesis and a premise. The relationship can be entailment, contradiction, or neutral, depending on the compatibility of the premise and hypothesis.

Both approaches you mentioned have their own advantages and disadvantages.

Multiple Samples Approach:

Advantages:

• Provides distinct training instances for each label, allowing the model to learn specific relationships between the sentence and individual labels.

Disadvantages:

• Data Redundancy: Generating multiple samples for the same sentence may lead to redundancy in the training data, potentially increasing the computational resources required for training without significant additional information gain
• Increased Training Time: The duplication of sentences for each label can lead to longer training times, especially for large datasets, as the model needs to process and learn from each duplicated sample.
• Potential Overfitting: The model may be more prone to overfitting on the training data due to the increased number of similar samples, which could impact its generalization to unseen data.

Single Sample Approach:

Advantages:

• Reduces data redundancy and training time by consolidating multiple labels into a single sample, potentially leading to more efficient model training.

Disadvantages:

• Loss of Label-Specific Information: By combining all labels into a single sample, the model may not effectively capture the nuanced relationships between the sentence and individual labels, potentially leading to lower classification accuracy for specific labels.
• Complex Relationship Representation: The model may struggle to learn complex relationships between the sentence and multiple labels within a single sample, potentially impacting its ability to accurately classify the input.
• Limited Label-Specific Training: The model may not receive sufficient training instances for each label, which could affect its ability to differentiate between different labels during classification.

I hope this helps you in evaluating the best approach for your project. If you have further questions or need additional insights, feel free to ask!