0
$\begingroup$

I am wondering, do we really need <unk> tokens? Why do we limit our vocabulary?

Is it for speed? Accuracy?

If we disable all limitations, what do you predict happens?

$\endgroup$
2
  • $\begingroup$ Please give a little more context. What are you using these tokens for? What do they replace? Any links to example usages or literature? $\endgroup$
    – n1k31t4
    Commented Jun 20, 2018 at 21:57
  • $\begingroup$ On sequence2sequence models a vocabulary is commonly used. This vocabulary though for some reason is limited and any word which is not in the vocabulary gets replaced with <unk>. $\endgroup$ Commented Jun 20, 2018 at 21:59

2 Answers 2

2
$\begingroup$

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the selection of a dictionary. If the word is not in the dictionary we have chosen, then we are saying we have no valid representation for that word (or we are simply not interested).

Other tags are commonly used to groups such thing together, not only (j)unk.

For example <EMOJI> might replace any token that is found in our list of deined emojis. We are keeping some information, i.e. that there is a symbol representing emotion of some kind, but we are neglecting exactly which emotion. You can think of many more examples where this might be helpful, or you just don't have the right (labelled) data to make the most of the contents semantically.

$\endgroup$
0
$\begingroup$

As it is already mentioned in the comments, in tokenizing and NLP when you see UNK token, it is to indicate unknown word with a hight chance.

for example, if you want to predict a missing word in a sentence. how would you feed your data to it? you definitely need a token for showing that where is the missing word. so if the "house" is our missing word, after tokenizing it will be like:

'my house is big' -> ['my', 'UNK', 'is', 'big']

$\endgroup$
1
  • $\begingroup$ This answer seems to be plagiarized from here $\endgroup$
    – Paradox
    Commented Jan 16, 2022 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.