2
$\begingroup$

in a text such as

"The deal with Canada's Barrick Gold finalised in Toronto over the weekend"

When I try to break it into bigram model, I get this

"The deal"
"deal with" 
"with Canada's"
"Canada's Barrick"
"Barrick Gold"
"Gold finalised"
"finalised in"
"in Toronto"
"Toronto over"
"over the"
"the weekend"

my question

shall I include the first word and the last word as single words

"* The"
"Weekend *"
Improve this question
$\endgroup$
2
  • $\begingroup$ Making an analogy with 2D convolutions used in computer vision, I would say you could, however I doubt here that this can improve the accuracy of your model so I would not do it. This is just my intuition to help you going. If you are not in a hurry, you can try both and compare the results. $\endgroup$
    – Jeanba
    Nov 18, 2019 at 10:44
  • $\begingroup$ What are you intending to do with the bigram model? If you need to be able to predict the first word of a sentence for instance, you will ideally have a simple way to model that. $\endgroup$
    – Neil Slater
    Nov 18, 2019 at 11:52

1 Answer 1

Reset to default
2
$\begingroup$

What you describe is called padding and is indeed used frequently in language modeling. For instance if one represents the sequence "A B C" with trigrams:

# # A
# A B
A B C
B C #
C # #

The advantages of padding:

  • it makes every word/symbol appear the same number of times whether it appears in the middle of the sequence or not.
  • it marks the beginning and end of a sentence/text, so that the model can represent the probability to start/end with a particular word.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.