I am wondering, do we really need <unk>
tokens?
Why do we limit our vocabulary?
Is it for speed? Accuracy?
If we disable all limitations, what do you predict happens?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this communityI am wondering, do we really need <unk>
tokens?
Why do we limit our vocabulary?
Is it for speed? Accuracy?
If we disable all limitations, what do you predict happens?
The <unk>
tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested).
Other tags are commonly used to groups such thing together, not only (j)unk.
For example <EMOJI>
might replace any token that is found in our list of deined emojis. We are keeping some information, i.e. that there is a symbol representing emotion of some kind, but we are neglecting exactly which emotion. You can think of many more examples where this might be helpful, or you just don't have the right (labelled) data to make the most of the contents semantically.
<unk>
. $\endgroup$ – A. Dandelion Jun 20 '18 at 21:59