# How to convert sequence of words in to numbers which are input to RNN/LSTM?

I am watching online videos and tutorials about use of RNN/LSTM for NLP but none of them explain how to convert the sequences of words into digitized input to the neural networks?

I am looking for intuitive understanding but answers with python code are also welcome.

For example, how do I input; "grass is green and sun is hot" to my RNN/LSTM?

• Could you just mark it as answered if you think that's the answer? Aug 29, 2019 at 16:01

from keras.preprocessing import Tokenizer
samples = ["grss is green and sun is hot"]
tokenizer = Tokenizer(num_words=1000)
tokenizer.fit_on_texts(samples)
sequences = tokenizer.texts_to_sequences(samples)


The Keras library uses it's tokenizer function but you have other well known libraries like nltk, gensim to convert them into sequences and pass it into your neural network. There are other ways like TF/IDF and CountVectorizer in Sklearn for more simpler algorithms. num_words takes the most 1000 frequent words and tokenizes them.

Danny is 100% correct. Also, different NLP problems require different preprocessing solutions.

how to convert the sequences of words into digitized input

First, Split the string text into unique words.

Then, I recommend ordering the outcome in descending order. Depending on the use case, you will probably want to remove or at least know which words repeat the most.

Finally, create a dictionary that maps the word to an integer in ascending order. For example, the most common expression would be index 0 (or 1 in some cases) and another dictionary to map the integers back to the words.

from collections import Counter

text = "the grass is green and the sun is hot"

# split the text into words
word_counts = Counter(text.split(' '))

# list the words from most common to less common
sorted_counts = sorted(word_counts, key=word_counts.get, reverse=True)

# creates an integer to word dictionary
int_to_word = {i: word for i, word in enumerate(sorted_counts)}

# creates a word to integer dictionary
word_to_int = {word: i for i, word in int_to_word.items()}

# convert the words to numbers
for word, i in word_to_int.items():
text = text.replace(word, f'{i}')


Integer to Word:

{0: 'the', 1: 'is', 2: 'grass', 3: 'green', 4: 'and', 5: 'sun', 6:> 'hot'}

Word to Integer:

{'the': 0, 'is': 1, 'grass': 2, 'green': 3, 'and': 4, 'sun': 5, 'hot': 6}

Finally,

text = '0 2 1 3 4 0 5 1 6'

Depending on the problem you're trying to solve, you will likely need additional processing (for example, the numbers are still strings). So there is a lot of work ahead of you. However, at least you now know how to represent words as numbers and decode numbers back to words!