# Why some ML models can't take advantage of text ordering information?

> Build n-gram model [Option A] We refer to models that process the tokens independently (not taking into account word order) as n-gram models. Simple multi-layer perceptrons (including logistic regression), gradient boosting machines and support vector machines models all fall under this category; they cannot leverage any information about text ordering. >

Then

> Build sequence model [Option B] We refer to models that can learn from the adjacency of tokens as sequence models. This includes CNN and RNN classes of models. Data is pre-processed as sequence vectors for these models >

Why some ML model (Simple multi-layer perceptrons, gradient boosting machines and support vector machines models) cannot leverage any information about text ordering?

Is it because they can't accept floats on their input? (Not sure if those ML models accept floats on their input. I am guessing they can't.)

• Please check the definition of n-gram. Only the words in the n-gram preserve the original order. The n-grams are unordered. – Marmite Bomber Jan 13 '19 at 20:35
• Sorry, I don't follow. We could have n_grams encoded with 1 hot encoder and use them on some other ML models. So, why they said some ML models cannot leverage any information about text ordering? I have just edited the question a bit more to clarify it. – Gabriel Jan 14 '19 at 18:47

This tutorial assumes the text will be represented as a fixed length feature vector. There are several ways to do that. The good old one is using n-grams, indexing and several tricks around that.

First, N-grams can be seen as kind of words (or rather, phrases). The extreme version of it is unigram, or 1-gram, which is just a single word. Bi-grams, tri-grams and so on encode certain word ordering, of course, but the ordering of the n-grams in the text is usually ignored. Let's assume you use unigrams and defined the vocabulary:

Index assigned for every token:
{'the': 0, 'mouse': 1, 'ran': 2, 'up': 3, 'clock': 4 }


By using the count encoding, we come up with the following vector representing the text:

'The mouse ran up the clock' = [2, 1, 1, 1, 1]


But if you simply shuffle the words, you will end up with the same vector.

2-grams, 3-grams and so on capture the word ordering better and it will be more difficult to get similar vectors for different texts. So, this is not completely true that they cannot leverage any information about text ordering. However, just by swapping sentences in a text, you can get very close representations. Especially, if some rare 3-grams are dropped from vocabulary (which is widely used to keep the vocabulary of reasonable size).

Recurrent neural networks and word embeddings are more modern approaches in NLP. In case of RNNs, words (in form of embeddings) are fed to the model sequentially (not independently like in case of n-grams), so the model is much more sensitive to the word ordering.

• Thanks! When you say "In case of RNNs, words (in form of embeddings) are fed to the model sequentially (not independently like in case of n-grams), so the model is much more sensitive to the word ordering.". Any link with an exampl where I can find more about that? Thanks again! – Gabriel Jan 15 '19 at 21:07
• Here is an example on sentiment analysis using LSTMs: machinelearningmastery.com/… – Dmytro Prylipko Jan 16 '19 at 7:55