How to feed data for ngram model?

Question

I want to train an ngram language model

Let's say I have the following corpus:

The sliding cat is not able to dance
He is only able to slide
Because obviously he is the sliding cat

I am planning to use tf.data.Dataset to feed my model, which is fine

But I don't know if it is better to use a sliding window to iterate through my copus or simply feed my corpus n words at a time

Using a sliding window, my model (assuming a bigram) will see:

The sliding
sliding cat
cat is
is not
...

Going n word at a time:

The sliding
cat is
not able
...

I'd appreciate any recommandation, thanks

Answer 1

You should definitely use a sliding window.

An n-gram language model represents the probabilities for all the n-grams. If it doesn't see a particular n-gram in the training data, for example "sliding cat", it will assume that this n-gram has probability zero (actually zero probabilities are usually replaced with very low probability by smoothing, in order to account for out-of-vocabulary n-grams). This would result in a zero probability for a sentence which was actually in the training case (or a very low probability with smoothing).

Also it's common to use "padding" at the beginning and end of every sentence, like this:

#SENT_START# The
The sliding
sliding cat
cat is
is not
...
to dance
dance #SENT_END#

This gives the model indications about the words more likely to be at the beginning or end (it also balances the number of n-grams by word in a sentence: exactly $n$ even for the first/last word).