in a text such as

"The deal with Canada's Barrick Gold finalised in Toronto over the weekend"

When I try to break it into bigram model, I get this

"The deal"
"deal with" 
"with Canada's"
"Canada's Barrick"
"Barrick Gold"
"Gold finalised"
"finalised in"
"in Toronto"
"Toronto over"
"over the"
"the weekend"

my question

shall I include the first word and the last word as single words

"* The"
"Weekend *"
  • $\begingroup$ Making an analogy with 2D convolutions used in computer vision, I would say you could, however I doubt here that this can improve the accuracy of your model so I would not do it. This is just my intuition to help you going. If you are not in a hurry, you can try both and compare the results. $\endgroup$
    – Jeanba
    Commented Nov 18, 2019 at 10:44
  • $\begingroup$ What are you intending to do with the bigram model? If you need to be able to predict the first word of a sentence for instance, you will ideally have a simple way to model that. $\endgroup$ Commented Nov 18, 2019 at 11:52

1 Answer 1


What you describe is called padding and is indeed used frequently in language modeling. For instance if one represents the sequence "A B C" with trigrams:

# # A
# A B
B C #
C # #

The advantages of padding:

  • it makes every word/symbol appear the same number of times whether it appears in the middle of the sequence or not.
  • it marks the beginning and end of a sentence/text, so that the model can represent the probability to start/end with a particular word.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.