Text classification for data with multiple labels per observation

Question

I have a dataset of tweets that has been labeled by multiple people. So the columns look something like:

Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc.

The idea is to build a tweet classifier based on the labels. How should I input this data into the classifier? I was thinking of taking the plurality response, but is there a way I can use all of the data?

Also, there is partial coverage, so not every labeler labeled every tweet.

Answer 1

Check this blog post about multi-label classification. Encode your labels using Multi Label Binarizer, then you optimize binary crossentropy loss over sigmoid on those encoded labels.

Answer 2

There are different options:

• Multi-class classification: each tweet can be labelled with several classes
• Majority voting: each tweet is labelled with the class chosen most often
• Consider tweets with (too many) different labels as ambiguous and remove them from the dataset
• ...

Additionally it's usually useful to calculate the inter-annotator agreement, because this indicates how subjective the task is and therefore how accurate one can expect the predictions to be.