Group/cluster semantically similar classes in reports?

Question

I'm fine-tuning BERT models to binary classify reports. For example, a report can be about 'birds' or not about 'birds'.

This works really well, but now I want to do multi-label classification, because I want to classify for about 1000 animals. However, some animals are closely related (for example: 'raven' and 'crow').

Predicting these individual labels will get lower accuracy, but if I would group them together in 'raven or crow' I get a much higher accurary.

However, I don't want to manually make these groups. Is there a generic method that would use the LLM and classes to create those groups for me?

Answer 1

I don't have any experience with using LLM's for this purpose, but you might be able to use embeddings to create this grouping yourself. Depending on the type/length of your labels you could use something like BERT or S-BERT to generate embeddings for your labels. You can then calculate the similarity between all embeddings (using cosine similarity or any other similarity measure), and group labels together if the similarity is above some threshold. The MTEB benchmark tests different models to see how they perform when generating embeddings for different tasks.