i'm kinda new with neural machine translation. I've read some papers, authors usually limit the size of vocabulary by replace rare words by unk token. In this paper, they said that "...NMT model cannot learn the translation of rare words...". I want to understand why is hard for NMT mode to learn rare words and also the impact of word counts to NMT model. Thanks
1 Answer
Rare words are not a problem only for NMT, they are a problem for MT in general. The reason is simple: in order to accurately translate a word in any particular context, the model needs to see as many examples as possible during the training stage. By definition the training data contains very few occurrences of rare words (especially hapax words which occur only once), so the model doesn't have enough information to learn their translation properly.