I am watching online videos and tutorials about use of RNN/LSTM for NLP but none of them explain how to convert the sequences of words into digitized input to the neural networks?

I am looking for intuitive understanding but answers with python code are also welcome.

For example, how do I input; "grass is green and sun is hot" to my RNN/LSTM?

  • $\begingroup$ Could you just mark it as answered if you think that's the answer? $\endgroup$
    – Danny
    Aug 29, 2019 at 16:01

2 Answers 2

from keras.preprocessing import Tokenizer
samples = ["grss is green and sun is hot"]
tokenizer = Tokenizer(num_words=1000)
sequences = tokenizer.texts_to_sequences(samples)

The Keras library uses it's tokenizer function but you have other well known libraries like nltk, gensim to convert them into sequences and pass it into your neural network. There are other ways like TF/IDF and CountVectorizer in Sklearn for more simpler algorithms. num_words takes the most 1000 frequent words and tokenizes them.

Link : Keras text processing


Danny is 100% correct. Also, different NLP problems require different preprocessing solutions.

Regarding your question:

how to convert the sequences of words into digitized input

First, Split the string text into unique words.

Then, I recommend ordering the outcome in descending order. Depending on the use case, you will probably want to remove or at least know which words repeat the most.

Finally, create a dictionary that maps the word to an integer in ascending order. For example, the most common expression would be index 0 (or 1 in some cases) and another dictionary to map the integers back to the words.

from collections import Counter  

text = "the grass is green and the sun is hot"

# split the text into words
word_counts = Counter(text.split(' '))

# list the words from most common to less common 
sorted_counts = sorted(word_counts, key=word_counts.get, reverse=True)

# creates an integer to word dictionary 
int_to_word = {i: word for i, word in enumerate(sorted_counts)} 

# creates a word to integer dictionary 
word_to_int = {word: i for i, word in int_to_word.items()} 

# convert the words to numbers
for word, i in word_to_int.items():
    text = text.replace(word, f'{i}')

Integer to Word:

{0: 'the', 1: 'is', 2: 'grass', 3: 'green', 4: 'and', 5: 'sun', 6:> 'hot'}

Word to Integer:

{'the': 0, 'is': 1, 'grass': 2, 'green': 3, 'and': 4, 'sun': 5, 'hot': 6}


text = '0 2 1 3 4 0 5 1 6'

Depending on the problem you're trying to solve, you will likely need additional processing (for example, the numbers are still strings). So there is a lot of work ahead of you. However, at least you now know how to represent words as numbers and decode numbers back to words!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.