I am creating an n-gram model that will predict the next word after an n-gram (probably unigram, bigram and trigram) as coursework.

I have seen lots of explanations about HOW to deal with zero probabilities for when an n-gram within the test data was not found in the training data. I understand how 'add-one' smoothing and some other techniques work.

However, I can find nothing about WHY we need to take actions such as these.

For instance, if the test data has "Peace begins with a Smile" and this was not in the training data, so when I supply the model with "Peace begins with a", it will not come up with "Smile" end word. It may provide others or none. If there are none or they have a low probability, then I would supply the shorter n-gram of "begins with a" and see what words and probabilities that provides. If that fails, then "with a" and so on.

I suspect I'm missing something but can't see what.

Please can you help?


The purpose of smoothing is to prevent a language model from assigning zero probability to unseen events.

That is needed because in some cases, words can appear in the same context, but they didn't in your train set. Smoothing is a quite rough trick to make your model more generalizable and realistic. You can also see it as a tool to prevent overfitting.

| improve this answer | |
  • $\begingroup$ I suggest you this great book on NLP. In particular, Chapter 3 on N-grams treats smoothing in detail. I found it very useful. $\endgroup$ – Leevo Jul 16 '19 at 7:51
  • $\begingroup$ Thanks @Leevo and yes it’s a great book. It has clarified that backoff (which I do get the purpose of) is a method of smoothing. I can see the add one method and others will avoid assigning a zero probability. But I still don’t get why having a zero probability is a problem. If that n-gram didn’t appear, even though the word is in the training corpus, then while we might not like it, the KNOWN probability of that n-gram IS zero. Using the add one method, wouldn’t you just get an equal small probability for all the words available in the corpus? I can’t yet see how that helps. $\endgroup$ – Chris Jul 16 '19 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.