Best way to tokenize tweet

Question

While working with Twitter datasets, one thing that always confuses me is, How to tokenize the tweets.

I have seen different open-source implementations using different schemes for tokenization. They handle URL-mentions, Capitalization, User-mention etc. differently.

I usually follow the script accompanying the GloVE code: https://nlp.stanford.edu/projects/glove/ .

Are there any std. rules / best practices one should follow while tokenizing tweets? So much variation in different code-bases confuses me sometimes.

Answer 1

Something that has helped me when I have worked with tweets is what is the idea behind using the tweets. There is no standard way of doing it as sometimes things that are irrelevant to tokenizers are the ones you actually need. I would take a generic approach of tokenizer, modified for the use case I work with.

Also, ensure you keep a copy of the raw tweets after processing, so that you do not lose any information, you might need it for later use.

Answer 2

Tokenization, like all preprocessing, is application specific. It depends on the end goal. The most difficult (and interesting) parts of tweets are usernames, hashtags, URLs, and emoticons. Different applications will model these elements differently. Thus the need for different tokenizers with different parameters.