Training a model purely on weak labels

Question

I have read a couple of papers now use rules-based system to create weak labels and then train a BERT-based model only using these weak labels. Both studies have reported better performances on manually labelled gold-standard test data.

However, I just don't follow the logic here. I understand distant supervision and all that. It's been around for a while now. I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as "ground-truth", and more importantly, don't you already know how to create "ground-truth" (by the rules-based system) ??? What's the point of the 2nd step?

Even though the performances on test data are better. It doesn't convince me your ML model (from the 2nd step) have learnt something beyond the weak labels you fed to it.

The only argument I find has some merits is that you basically treat the BERT-based model already as a zero-shot classifier and you are fine-tuning it with weak labels. I am just confused. Can someone pls enlighten me? Am I missing something obvious here?

Answer 1

Based on a quick read of the paper linked in the comment:

I just don't understand if your model (BERT or not) is only trained on these weak labels then you are treating them as "ground-truth",

Correct, but only for the training: the model is trained to recognize the labels obtained with a "quick and dirty" method.

and more importantly, don't you already know how to create "ground-truth" (by the rules-based system) ??? What's the point of the 2nd step?

No, because the real ground truth they are interested in is not those from the "quick and dirty" method. If they were, it would indeed be sufficient to run their rule-based system. The goal is to predict the labels obtained in what the authors call the "Gold Standard Corpus", which was manually annotated and never seen by the model.

Typically the quick and dirty method will result in some classification errors. The point of tuning the model with these labels is to see if the model can extrapolate from these low-quality labels to high-quality labels. This ability to generalize beyond the training data is based on the underlying semantic information contained in the original BERT-like model. For example this model might be able to associate a specific sport like swimming with "physical activity", even though the weak supervision doesn't contain this association.