0
$\begingroup$

I want to train an ngram language model

Let's say I have the following corpus:

The sliding cat is not able to dance
He is only able to slide
Because obviously he is the sliding cat

I am planning to use tf.data.Dataset to feed my model, which is fine

But I don't know if it is better to use a sliding window to iterate through my copus or simply feed my corpus n words at a time

Using a sliding window, my model (assuming a bigram) will see:

The sliding
sliding cat
cat is
is not
...

Going n word at a time:

The sliding
cat is
not able
...

I'd appreciate any recommandation, thanks

$\endgroup$
0
$\begingroup$

You should definitely use a sliding window.

An n-gram language model represents the probabilities for all the n-grams. If it doesn't see a particular n-gram in the training data, for example "sliding cat", it will assume that this n-gram has probability zero (actually zero probabilities are usually replaced with very low probability by smoothing, in order to account for out-of-vocabulary n-grams). This would result in a zero probability for a sentence which was actually in the training case (or a very low probability with smoothing).

Also it's common to use "padding" at the beginning and end of every sentence, like this:

#SENT_START# The
The sliding
sliding cat
cat is
is not
...
to dance
dance #SENT_END#

This gives the model indications about the words more likely to be at the beginning or end (it also balances the number of n-grams by word in a sentence: exactly $n$ even for the first/last word).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.