4
$\begingroup$

Is it real to use word's n-grams for Deep Neural Network?

E.g., sentences list contains in X_train dataframe with "sentences" column. I use tokenizer from Keras in the next manner:

tokenizer = Tokenizer(lower=True, split=' ')
tokenizer.fit_on_texts(X_train.sentences)
X_train_tokenized = tokenizer.texts_to_sequences(X_train.sentences)

And later I use padding:

X_train_sequence = sequence.pad_sequences(X_train_tokenized)

Also I use simple LSTM network:

model = Sequential()
model.add(Embedding(MAX_FEATURES, 128))
model.add(LSTM(32, dropout=0.2, recurrent_dropout=0.2,
               activation='tanh', return_sequences=True))
model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2, activation='tanh'))
model.add(Dense(number_classes, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer = 'rmsprop',
              metrics=['accuracy'])

In this case, tokenizer execution. In Keras docs: https://keras.io/preprocessing/text/ I see character processing only, but it is nt apprepriate for my case.

My main question: Can I use n-grams for NLP tasks with deep learning (not necessary Sentiment Analysis, any abstract NLP task).

Indeed, in many tutorials or books I doesn't see any remainder n-grams for text processing, only embeddings.

For clarification: I'd like to consider not just words, but combination of words - I'd like to try it for my task.

$\endgroup$
  • 2
    $\begingroup$ Yes, you can. I recall seeing a paper on this but I can't find it right now. You can use feature hashing on the n-gram part to reduce the dimensionality. n-grams are also used to find embeddings, as in fastText (char n-grams) and word2vec (word n-grams). $\endgroup$ – Emre Sep 13 '17 at 19:06
3
$\begingroup$

Theoretically, you could use n-grams to model text sequences. But there are some good reasons why it's not often mentioned in textbooks.

Intuitively, modeling sequences is learning that after a, b, and c comes d. So if your sequence is:

"Hello, my name is Bob, what a lovely day today, how are you?"

It technically doesn't matter if a = "Hello", or "H", or "Hello, my name is Bob". To your model, it will just learn that after a thing, comes another thing etc.

So yes, you could use n-grams. But the problem is that you need to make sure that the way the ngrams are split is meaningful (i.e. "Hello, my" is a probably worse split than "my name is", which probably happens more often)

But don't worry, the whole point of using LSTM is that the memory part of the network will mimic using n-grams, as long as it helps with the overall performance. This means that if it needs to take more words into account to predict the next one, it will do it up to the depth of your network.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.