1
$\begingroup$

I have a dataset of tweets that has been labeled by multiple people. So the columns look something like:

Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc.

The idea is to build a tweet classifier based on the labels. How should I input this data into the classifier? I was thinking of taking the plurality response, but is there a way I can use all of the data?

Also, there is partial coverage, so not every labeler labeled every tweet.

$\endgroup$
0
$\begingroup$

There are different options:

  • Multi-class classification: each tweet can be labelled with several classes
  • Majority voting: each tweet is labelled with the class chosen most often
  • Consider tweets with (too many) different labels as ambiguous and remove them from the dataset
  • ...

Additionally it's usually useful to calculate the inter-annotator agreement, because this indicates how subjective the task is and therefore how accurate one can expect the predictions to be.

$\endgroup$
0
$\begingroup$

Check this blog post about multi-label classification. Encode your labels using Multi Label Binarizer, then you optimize binary crossentropy loss over sigmoid on those encoded labels.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.