Why would a sequence-model vs n-gram model depending on ratio of Samples / Words per Sample

Question

This ML tutorial from Google is analyzing the imdb reviews dataset to predict the tag positive or negative. When choosing a model

1. Calculate the number of samples/number of words per sample ratio.
2. If this ratio is less than 1500, tokenize the text as n-grams and use a simple multi-layer perceptron (MLP) model to classify them
3. If the ratio is greater than 1500, tokenize the text as sequences and use a sepCNN model to classify them (right branch in the flowchart below):

Let's call the ratio "Number of Samples / Words Per Sample" as "S/W":

Later the tutorial says:

When the S/W ratio is small, we’ve found that n-gram models perform better than sequence models.

I'm trying to think of an intuitive explanation of why that the above is true. The tutorial says:

Sequence models are better when there are a large number of small, dense vectors. This is because embedding relationships are learned in dense space, and this happens best over many samples.

Now it's important to say when making the decision based on "S/W", the Google tutorial was not just trying to maximize accuracy, but also...

... optimizing for the best accuracy that could be achieved in the shortest possible compute time

So I'm keeping the "balance high accuracy with low compute time" in mind when trying to understand:

• why a small S/W can use an n-gram model
• and a large S/W can use a sequence model

In particular I'm trying to understand in relation to numerator and denominator, say...

• with a fixed W (words per sample), a larger numerator S (number of samples), would increase the ratio S/W, and therefore push us toward a sequence model
• with a fixed S (number of samples), a smaller denominator W (words per sample) would also increase the ratio S/W, and therefore push us toward a sequence model

Please note I am not asking why MLP's, SGB's, SVM's can't handle "sequence vectors" -- there is already a question for that, and it's a good question, but I'm asking why you would choose to use n-gram approach vs sequence approach based on the ratio.

Answer 1

---

I guess the tutorial answered my question, and I quoted the answer, I just need to really wrap my head around it:

Sequence models are better when there are a large number of small, dense vectors. This is because embedding relationships are learned in dense space, and this happens best over many samples.

The n-gram approach is also called a "bag of words approach", and doesn't communicate anything about word order to the model.

With n-gram vector representation, we discard a lot of information about word order and grammar (at best, we can maintain some partial ordering information when n > 1). This is called a bag-of-words approach.

Also, the sequence model approach uses a neural network; the neural network contains a hidden layer that will learn "embeddings" (word similarity)

Sequence models often have such an embedding layer as their first layer

So why would a large S/W ratio push us toward a sequence model approach?

As a rule I think neural nets do best with large amounts of data; so the larger the numerator (number of samples), the better we can do with a neural net (RNN specifically recommended for the sequence model, vs a MLP recommended for the n-gram/bag-of-words model)

And as the tutorial explained, the more "dense" the samples, (aka the smaller the numerator (number of words per sample)), the better chance the neural net has at "understanding word similarity" (embeddings).

Intuitively, I guess that makes sense to my naive human brain; these "dense" sentences help me learn word association better...

['Ascend to go up',
 'Descend to go down']

But maybe I should consider other things,

• like the type of document/writing (the examples I gave above are very deliberate & "instructional" in tone)
• or the size of the vocabulary

And maybe more than anything, the answer "why" is simply because "they tested different approaches based on the ratio" (a kind of hyper-hyper tuning!), in other words, they tested a lot, found the ratio is a meaningful "dataset parameter", which eventually passes "threshhold" in the tradeoff of model accuracy + performance time.

We ran a large number (~450K) of experiments across problems of different types (especially sentiment analysis and topic classification problems), using 12 datasets, alternating for each dataset between different data preprocessing techniques and different model architectures. This helped us identify dataset parameters that influence optimal choices.