This ML tutorial from Google is analyzing the imdb reviews dataset to predict the tag positive or negative. When choosing a model
- Calculate the number of samples/number of words per sample ratio.
- If this ratio is less than 1500, tokenize the text as n-grams and use a simple multi-layer perceptron (MLP) model to classify them
- If the ratio is greater than 1500, tokenize the text as sequences and use a sepCNN model to classify them (right branch in the flowchart below):
Let's call the ratio "Number of Samples / Words Per Sample" as "S/W":
Later the tutorial says:
When the S/W ratio is small, we’ve found that n-gram models perform better than sequence models.
I'm trying to think of an intuitive explanation of why that the above is true. The tutorial says:
Sequence models are better when there are a large number of small, dense vectors. This is because embedding relationships are learned in dense space, and this happens best over many samples.
Now it's important to say when making the decision based on "S/W", the Google tutorial was not just trying to maximize accuracy, but also...
... optimizing for the best accuracy that could be achieved in the shortest possible compute time
So I'm keeping the "balance high accuracy with low compute time" in mind when trying to understand:
- why a small S/W can use an n-gram model
- and a large S/W can use a sequence model
In particular I'm trying to understand in relation to numerator and denominator, say...
- with a fixed W (words per sample), a larger numerator S (number of samples), would increase the ratio S/W, and therefore push us toward a sequence model
- with a fixed S (number of samples), a smaller denominator W (words per sample) would also increase the ratio S/W, and therefore push us toward a sequence model
Please note I am not asking why MLP's, SGB's, SVM's can't handle "sequence vectors" -- there is already a question for that, and it's a good question, but I'm asking why you would choose to use n-gram approach vs sequence approach based on the ratio.