I am trying to create a sentiment analysis program which will classify some of the tweets which i have collected under a hashtag. There are 7750 tweets in the dataset and I am labeling them into the two classes now. Then I will use a neural network to classify them into positive and negative classes.

The problem with the dataset is that it posses a lot of redundant data (Retweets basically). Manually deleting them is not possible and I have tried to find a programmable solution via the Tweepy API but couldn't find any.

So my question is do I need to get rid of these redundant records or should i leave them as is?


1 Answer 1


Assuming you want to learn sentiment this is a problem. What happens when you feed this to a Machine Learning algorithm is that it will give more weight to the tweets that are in there multiple times, while not learning more information. It's unlikely that a tweet that has been retweeted 10 times carries more significant information about sentiment than one that hasn't been retweeted. What you could do is add the number of times the tweet was in the set as a feature to your sentiment model, it's possible something can be learned from that fact (maybe positive tweets are retweeted more often), but you should keep it at 1 row for every distinct tweet.

Getting rid of these redundant records should not be difficult programmatically though. I don't know what language you are using but if you only consider the body of the tweet (the content) you could iterate over your tweets, keep a list of all the unique bodies, combined with other meta information (like user, labeled sentiment) and if the content of the next tweet is already in there, just do not add it. Look for 'distinct' functionality as opposed to redundant, enough information out there.

  • 1
    $\begingroup$ I am using python, I like your idea, its not that i haven't thought about it. People have some more characters( 'RT' and the handle of an account etc). I could preprocess the data while removing RT and the account handles. I think i will try that now and get back to you. $\endgroup$
    – Bhashithe
    Commented Jun 5, 2016 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.