1
$\begingroup$

On this google tutorial (https://developers.google.com/machine-learning/guides/text-classification/step-4) it is said:

> Build n-gram model [Option A] We refer to models that process the tokens independently (not taking into account word order) as n-gram models. Simple multi-layer perceptrons (including logistic regression), gradient boosting machines and support vector machines models all fall under this category; they cannot leverage any information about text ordering. >

Then

> Build sequence model [Option B] We refer to models that can learn from the adjacency of tokens as sequence models. This includes CNN and RNN classes of models. Data is pre-processed as sequence vectors for these models >

Why some ML model (Simple multi-layer perceptrons, gradient boosting machines and support vector machines models) cannot leverage any information about text ordering?

Is it because they can't accept floats on their input? (Not sure if those ML models accept floats on their input. I am guessing they can't.)

$\endgroup$
2
  • 1
    $\begingroup$ Please check the definition of n-gram. Only the words in the n-gram preserve the original order. The n-grams are unordered. $\endgroup$ Commented Jan 13, 2019 at 20:35
  • 1
    $\begingroup$ Sorry, I don't follow. We could have n_grams encoded with 1 hot encoder and use them on some other ML models. So, why they said some ML models cannot leverage any information about text ordering? I have just edited the question a bit more to clarify it. $\endgroup$
    – Gabriel
    Commented Jan 14, 2019 at 18:47

2 Answers 2

2
$\begingroup$

This tutorial assumes the text will be represented as a fixed length feature vector. There are several ways to do that. The good old one is using n-grams, indexing and several tricks around that.

First, N-grams can be seen as kind of words (or rather, phrases). The extreme version of it is unigram, or 1-gram, which is just a single word. Bi-grams, tri-grams and so on encode certain word ordering, of course, but the ordering of the n-grams in the text is usually ignored. Let's assume you use unigrams and defined the vocabulary:

Index assigned for every token: 
{'the': 0, 'mouse': 1, 'ran': 2, 'up': 3, 'clock': 4 }

By using the count encoding, we come up with the following vector representing the text:

'The mouse ran up the clock' = [2, 1, 1, 1, 1]

But if you simply shuffle the words, you will end up with the same vector.

2-grams, 3-grams and so on capture the word ordering better and it will be more difficult to get similar vectors for different texts. So, this is not completely true that they cannot leverage any information about text ordering. However, just by swapping sentences in a text, you can get very close representations. Especially, if some rare 3-grams are dropped from vocabulary (which is widely used to keep the vocabulary of reasonable size).

Recurrent neural networks and word embeddings are more modern approaches in NLP. In case of RNNs, words (in form of embeddings) are fed to the model sequentially (not independently like in case of n-grams), so the model is much more sensitive to the word ordering.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks! When you say "In case of RNNs, words (in form of embeddings) are fed to the model sequentially (not independently like in case of n-grams), so the model is much more sensitive to the word ordering.". Any link with an exampl where I can find more about that? Thanks again! $\endgroup$
    – Gabriel
    Commented Jan 15, 2019 at 21:07
  • 2
    $\begingroup$ Here is an example on sentiment analysis using LSTMs: machinelearningmastery.com/… $\endgroup$ Commented Jan 16, 2019 at 7:55
1
$\begingroup$

The accepted answer is good, but I wanted more.

First I should say this is a good guess, but it's not correct AFAIK:

Is it because they can't accept floats on their input?

  • Both types of data preprocessing (n-gram data prep and sequence data prep) can produce integers and/or floats.
  • Both types of models (MLP's and SVM's, as well as RNN's), can accept integers and/or floats.

Here's what I learned about sequences:

  • when preparing the data, only certain preprocessing preserves order information
  • when choosing a model, only certain models can understand order information

When preparing the data...

for the sequence model

Note this example, which is how the Google tutorial prepares data for the sequence model (where order matters):

from tensorflow.keras.preprocessing import text
from tensorflow.keras.preprocessing import sequence

ts = ['The mouse ran up the clock', 'The mouse ran down']

tokenizer = text.Tokenizer(num_words=10)
tokenizer.fit_on_texts(ts)
sequence.pad_sequences(tokenizer.texts_to_sequences(ts), 6)

array([[1, 2, 3, 4, 1, 5],
       [0, 0, 1, 2, 3, 6]], dtype=int32)

... as shown in Google's image:... showing how a text_to_sequences preserves order, and becomes an embedding layer

for the bag-of-words aka n-gram model..

Note this example, which is how the Google tutorial prepares its data for the n-gram model (although I use unigrams); unigrams don't remember anything about word order. Notice these sentences which have different order (resulting in opposite meaning), appear to be the same as far as TfidfVectorizer is concerned; the two rows are identical:

from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()

ts = ['Cat beats mouse', 'Mouse beats cat']
print(v.fit_transform(ts).todense())
[[0.57735027 0.57735027 0.57735027]
 [0.57735027 0.57735027 0.57735027]]

As @Dmytro / accepted answer says, we could use bigrams (n-gram of size 2), or increasing-size-grams. The effect is that the two different sentences (different only in word order) are encoded differently (so there is some understanding these sentences have different meaning), as shown in the example below (the two rows are not identical)

from sklearn.feature_extraction.text import TfidfVectorizer
# Notice now we allow bi-grams ----v
v = TfidfVectorizer(ngram_range=(1,2))

ts = ['Cat beats mouse', 'Mouse beats cat']
print(v.fit_transform(ts).todense())
# Notice the bigram "beats mouse" is different than "beats cat", 
# each of those bigrams appear in only one of the rows, making the rows' meaning distinct
[[0.37930349 0.         0.53309782 0.37930349 0.53309782 0.37930349
  0.        ]
 [0.37930349 0.53309782 0.         0.37930349 0.         0.37930349
  0.53309782]]

However, increasing n-gram size won't remember the entire sentence order (unlike the text_to_sequence example above).

Moreover, increasing n-gram size produces increasingly rare combinations that you may discard from your vocabulary during feature selection (because your vocabulary; the count of distinct n-grams) is too large, as @Dmytro says.

Therefore only certain types of preprocessing will preserve order in the first place! If you don't preserve order, it doesn't matter what model you choose, there is no order to learn from!

When choosing a model

This is your main question:

Why some ML model (Simple multi-layer perceptrons, gradient boosting machines and support vector machines models) cannot leverage any information about text ordering?

models that don't understand order can still try to use the data...

Could you pass the texts_to_sequences data into an MLP model? Sure! It's a matrix. Revisit the example above; this array can be passed into an MLP model (or an SVM model, or even other types of model).

# ['The mouse ran up the clock', 'The mouse ran down']
array([[1, 2, 3, 4, 1, 5],
       [0, 0, 1, 2, 3, 6]], dtype=int32)

This is where I struggled with @Dmytro's language

... In case of RNNs, words (in form of embeddings) are fed to the model sequentially ...

In the phrase "words ... are fed to", I think "words" refers to the data in the matrix matrix; but the data would be "fed to" an MLP the same way it would be fed to an "RNN". The key difference is that MLP interprets each column as a separate feature; (as @Dmytro said; interprets feature values "independently"); so it doesn't learn "word 1 is often followed by word 2, then word 3, word 4, etc.", as represented in the first row:

[1, 2, 3, 4, 1, 5]

In fact MLP may even learn correlations between features (when the first column is "1" the second column is "2"), but what about a shift/padding for a simpler sentence 'The mouse ran up'. Now it's the third column with word "1" and the fourth column with word "2":

[0, 0, 1, 2, 3, 4]

But only certain models (like RNN's) are able to understand order...

The special thing about RNN's is that they can learn order within a row; they're good at sequences.

(Conversely, if you don't care about order, you probably don't need an RNN)

And why is it that RNN's "can do order" whereas other models like MLP cannot? Because RNN's have "memory"; they use backpropagation. This means the same row can be reprocessed in different time steps. At the first timestep , the first word is processed (red), and the output of that processing can be considered at the second timestep, when the second word is processed (blue)

a diagram of a recurrent neural network vs a feedforward neural network

For me this helps clarify how the architecture of RNN is specifically designed for a "memory" across the "order" of feature values within a row, whereas other simpler models don't work that way (and thus can only understand "order" if it is encoded in new features (bi-grams, tri-grams, etc))

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.