I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After creating a dictionary, I'm running a script in Python to replace strings (words) with numbers that keras/theano will embed later.
The problem is that such a large dataset requires a lot of time for lookup. I would appreciate if anyone had suggestion on a tool for faster lookup or similar. Currently I just loop through every word in the corpus and replace it with the corresponding number from the dictionary (1-hot encoding essentially)
I'm doing roughly the following: each Python list is a sentence (before tokenization here):
which I want to conver to a list of integers, like:
I referred to it (probably incorrectly) as one-hot encoding because for each word there is exactly one number in the dictionary.