Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am watching online videos and tutorials about use of RNN/LSTM for NLP but none of them explain how to convert the sequences of words into digitized input to the neural networks?
I am looking for intuitive understanding but answers with python code are also welcome.
For example, how do I input; "grass is green and sun is hot" to my RNN/LSTM?
from keras.preprocessing import Tokenizer
samples = ["grss is green and sun is hot"]
tokenizer = Tokenizer(num_words=1000)
tokenizer.fit_on_texts(samples)
sequences = tokenizer.texts_to_sequences(samples)
The Keras library uses it's tokenizer function but you have other well known libraries like nltk, gensim to convert them into sequences and pass it into your neural network. There are other ways like TF/IDF and CountVectorizer in Sklearn for more simpler algorithms. num_words takes the most 1000 frequent words and tokenizes them.
Link : Keras text processing
Danny is 100% correct. Also, different NLP problems require different preprocessing solutions.
Regarding your question:
how to convert the sequences of words into digitized input
First, Split the string text into unique words.
Then, I recommend ordering the outcome in descending order. Depending on the use case, you will probably want to remove or at least know which words repeat the most.
Finally, create a dictionary that maps the word to an integer in ascending order. For example, the most common expression would be index 0 (or 1 in some cases) and another dictionary to map the integers back to the words.
from collections import Counter
text = "the grass is green and the sun is hot"
# split the text into words
word_counts = Counter(text.split(' '))
# list the words from most common to less common
sorted_counts = sorted(word_counts, key=word_counts.get, reverse=True)
# creates an integer to word dictionary
int_to_word = {i: word for i, word in enumerate(sorted_counts)}
# creates a word to integer dictionary
word_to_int = {word: i for i, word in int_to_word.items()}
# convert the words to numbers
for word, i in word_to_int.items():
text = text.replace(word, f'{i}')
Integer to Word:
{0: 'the', 1: 'is', 2: 'grass', 3: 'green', 4: 'and', 5: 'sun', 6:> 'hot'}
Word to Integer:
{'the': 0, 'is': 1, 'grass': 2, 'green': 3, 'and': 4, 'sun': 5, 'hot': 6}
Finally,
text = '0 2 1 3 4 0 5 1 6'
Depending on the problem you're trying to solve, you will likely need additional processing (for example, the numbers are still strings). So there is a lot of work ahead of you. However, at least you now know how to represent words as numbers and decode numbers back to words!
