I wanted to work on twitter sentiment analysis.so before that I decided to collect some twitter data and label them on my own (pos,neg,neu). My doubt is that Should I clean the data before I label (i.e., removing RT,#,https,@ symbols) Or I can label them with out cleaning the Data? Does cleaning of data before labeling make any difference?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
The OP asks "Does cleaning of data before labeling make any difference?" - that's an empirical question... one that should be investigated by EDA of your data.
In some cases twitter-specific conventions can be highly indicative of a specific class (e.g., http:// associated with spam/advertising tweets) or sentiment (e.g., :-) emoticon associated with positive valence). Similarly, as I discuss in this post, stop words can be great features to keep in certain types of text models. My support for the above answer is based on an (unpublished) project I did that involved about 10 people hand-scoring 50k tweets stratified across major industries (food, travel, electronics, CPG, etc.) and brands (McDonalds, Southwest Airlines, iPhone, Tide PODs).
My advice, create feature extractors for any textual feature that you believe has a theoretical or logical justification for being good indicator of text polarity. Then empirically test each feature to determine whether it significantly improves classification accuracy. Keep those that are in your model; save the other feature extractors for a rainy day.
answered Jun 30, 2016 at 12:14
-
$\begingroup$ can i get that that 50k tweets data?@Brandon Loudermilk $\endgroup$ Commented Jun 30, 2016 at 12:29
-
$\begingroup$ I think point #5 from the linked post is really important. A lot of people want to perform "sentiment analysis" on data, but that's really just throwing around a term. @DilipBobby you should ask yourself why you want to perform sentiment analysis - what meaning do you want from this analysis - and then prepare the kind of data you need. $\endgroup$ Commented Jun 30, 2016 at 12:56
-
$\begingroup$ @DilipBobby - the 50k corpus represents a sizable investment in time and money; it is unlikely that my previous employer will readily share this IP. You should check on opendata.stackexchange.com for twitter data sources, like this one: opendata.stackexchange.com/questions/1545/twitter-open-datasets $\endgroup$ Commented Jun 30, 2016 at 13:08