1
$\begingroup$

I am working in emotion analysis of tweets. I collected close to .6 million tweets. I dont want to label them manually, instead i coded a bunch of complex rules to arrive at their label.

Before I use these algo generated labels for my ML solutions, I want to check/analyze how good is this labeling. Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.

$\endgroup$
2
$\begingroup$

Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.

Yes. You can treat your automated labelling as a model in its own right (it essentially is an "expert" model, that takes in additional features). Collect ground truth data with known accurate labels, and use a metric such as accuracy, AUROC, F1 score to determine how well your expert model works. Usually you should pick one metric that makes sense to you based on the problem you are trying to solve. Most statistics libraries will allow you to assess these metrics on arbitrary data, so it should be possible to find the relevant part of the API and feed it your synthetic labels to compare with ground truth data you have collected.

But wait! That is exactly what you would need to do in order to train and assess an ML model. About the best you can hope is that you will need less ground truth data to assess your expert than you would to train the ML version from scratch. For complex ML models such as deep neural networks, this could well be true.

Note that you may hit a limitation from this approach. Say that you assess from ground truth data that your expert is 85% accurate. You will not be able to use it to train an ML solution that is better than 85% accurate, because the ML will learn to copy the expert's predictions as best it can.

One thing to watch out for - if you analyse why the expert got some values wrong and then adjust it to give correct answers for those labels, you will need to remove the labels you used to do this from your test data - and ideally collect more, or take more from some unused pool, to replace them. Otherwise you will fit your expert to the test data and get incorrect reporting of how well the expert generalises to unseen data. Same reasoning means you should not use tweets and labels that you analysed when developing your expert model in order to test it.

$\endgroup$
  • $\begingroup$ Neil: your answer is incorrect. "Expert" uses features which are not available to ML algos. (one such feature is replies to a tweet. Expert uses this to be more sure of the label it gives, while ML algo never has access to any reply(ies) to a given tweet) $\endgroup$ – Anuj Gupta Sep 21 '16 at 11:42
  • $\begingroup$ @AnujGupta: That is worth adding to your question - that your rules include access to extra features. Still the part "You can collect ground truth data with known accurate labels, and use a metric such as accuracy, AUROC, F1 score to determine how well any model works." applies to you assessing your expert - the only way to assess the expert is to compare it with ground truth data. $\endgroup$ – Neil Slater Sep 21 '16 at 11:45
  • $\begingroup$ @AnujGupta: Updated my answer to take account of the new information. $\endgroup$ – Neil Slater Sep 21 '16 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.