I'm training an LSTM for sentiment analysis on a review dataset downloaded from here. The music review dataset contains about 150K data points (reviews of varying length labelled pos or neg). After creating a dictionary, I'm running a script in Python to replace strings (words) with numbers that keras/theano will embed later.

The problem is that such a large dataset requires a lot of time for lookup. I would appreciate if anyone had suggestion on a tool for faster lookup or similar. Currently I just loop through every word in the corpus and replace it with the corresponding number from the dictionary (1-hot encoding essentially)


I'm doing roughly the following: each Python list is a sentence (before tokenization here):

['noble', 'interesting_superlatives',...,'the_idea']

which I want to conver to a list of integers, like:

[143599, 12387,...,7582]

I referred to it (probably incorrectly) as one-hot encoding because for each word there is exactly one number in the dictionary.

  • $\begingroup$ It's not clear why do you need to look up in dictionary for "1-hot encoding"... Could you provide a small sample data set and desired data set based on that sample one? $\endgroup$ – MaxU Jan 16 '17 at 17:12
  • $\begingroup$ OK, I added an edit, hop it helps. $\endgroup$ – Alex Jan 16 '17 at 17:35
  • $\begingroup$ Querying dictionaries should be fast; are you not using a dict? See also feature hashing. $\endgroup$ – Emre Jan 16 '17 at 18:47
  • $\begingroup$ well there are 100K+ words in the dict (yes, I use it), 1 sentence takes ~1 sec, hence... $\endgroup$ – Alex Jan 16 '17 at 20:16

I'd like to extend the great @Emre's answer with another example - we are going to replace all tokenized words from the "1984" (c) George Orwell (120K words):

In [163]: %paste
import requests
import nltk
import pandas as pd

# source: https://github.com/dwyl/english-words
fn = r'D:\temp\.data\words.txt'
url = 'http://gutenberg.net.au/ebooks01/0100021.txt'

r = requests.get(url)

# read words into Pandas DataFrame
df = pd.read_csv(fn, header=None, names=['word'])
# shuffle DF, so we will have random indexes
df = df.sample(frac=1)
# convert Pandas DF into dictionary: {'word1': unique_number1, 'word2': unique_number2, ...}
lkp = df.reset_index().set_index('word')['index'].to_dict()

# tokenize "1984" (c) George Orwell
words = nltk.tokenize.word_tokenize(r.text)

print('Word Dictionary size: {}'.format(len(lkp)))
print('We have tokenized {} words...'.format(len(words)))
## -- End pasted text --
Word Dictionary size: 354983
We have tokenized 120251 words...

In [164]: %timeit [lkp.get(w, 0) for w in words]
10 loops, best of 3: 66.3 ms per loop

Conclusion: it took 66 ms to build a list of numbers for the list with 120K words from the dictionary containing 354.983 entries.

  • $\begingroup$ thanks, never used pandas before, for some reason the dictionary is created for phrases, so len(lkp) is 8300. Also, this doesn't really solve the problem: I need an array of numbers, where each number is an index from lkp. So I need to loop through every word to replace it with a number. I don't see how your script makes this solution faster. $\endgroup$ – Alex Jan 17 '17 at 12:09
  • $\begingroup$ @Alex, So I need to loop through every word to replace it with a number - this is exactly what my script is doing... Could you post a small reproducible data set (3-5 rows in the same format as your input data set) and desired data set (array / list / etc.)? $\endgroup$ – MaxU Jan 17 '17 at 12:12
  • $\begingroup$ Oh I'm sorry, just realized I did a wrong thing. It works really well. Not sure what's the trick though. I also use dict to store the dictionary with keys. Perhaps because I use a list. Need to look into it. Thanks. $\endgroup$ – Alex Jan 17 '17 at 13:53

You're doing something wrong. I can query a 100K word dict in nanoseconds

word_list = open('/usr/share/dict/words').read().split()

> 99171

word_dict = {word: hash(word) for word in word_list}
%timeit word_dict['blazing']

> 10000000 loops, best of 3: 33.8 ns per loop
  • $\begingroup$ same here, time.time() returned 0.576375007629 sec. But it's not what I need. I have a corpus. ~150K reviews, 20-30 words each. Each word in each review is replaced by a corresponding number from the dictionary. That's what taking ages. $\endgroup$ – Alex Jan 17 '17 at 11:11
  • $\begingroup$ How is that different from what I benchmarked? Multiply 33ns by 20-30 and you get 1ms, not 1s. Try using perf_counter instead of time. $\endgroup$ – Emre Jan 17 '17 at 16:40

You could use a trie from the Wikipedia definition:

is a kind of search tree—an ordered tree data structure that is used to store a dynamic set or associative array where the keys are usually strings.

pygtrie offers an implementation of tries with a dict interface. Here goes an example

import pygtrie as trie

words = ['cat', 'caterpillar', 'dog', 'mouse']

structure = trie.Trie()

for i, word in enumerate(words):
   structure[word] = i

print structure['caterpillar']

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.