I know that tokenizers turn words into numerics but what about hashtags? Are tokenizers design to handle hashtags or should I be filtering the "#" prior to tokenizing? What about the "@" symbol?
$\begingroup$
$\endgroup$
asked Mar 16, 2018 at 14:55
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using.
Consider this example tweet:
Hi, we need you!
#Hi #Weneedyou
If you use TreeBank or WordPunct tokenizers the output will be:
['Hi', 'we', 'need', 'you', '!', '#', 'Hi', '#', 'Weneedyou']
However if you use Whitespace Tokenizer, the result is:
['Hi', 'we', 'need', 'you!', '#Hi', '#Weneedyou']
Similar discrepancies can be found in terms such as can't or pre-order for instance.
Additionally, you need to consider what is a token for your task at hand. In my previous example Hi make sense either with or without #. However Weneedyou without the hash is just a poorly written word. Maybe you want to keep the hashtags in your corpus or maybe not, but this depends on what you want to do with it.
So first you need to know how your tokenizer handles these cases and decide if you want to remove them beforehand or later to keep hashtags in your corpus or only (sometimes weird) words.
In the case of the @ is exactly the same, you can keep it, remove it or maybe delete the whole instance @user as you don't want to keep user names in your corpus. As I said, it all depends on your task.
PS: In case you want to play around with different tokenizers, try this.
