0
$\begingroup$

I am trying to get the hash values for some English words from the string store of SpaCy.

nlp = English()
doc1 = nlp('this is doc1')
id = doc1.vocab.strings['Saurabh']
print(id)
# output, it has given hash code

id1 = doc1.vocab.strings['समाचार'] 
# This is a hindi word so i do not expect it to be a part of English,
# so it shoud throw an error.
print(id1)
# output, it given hash code

Why this is not giving an error in (at least) the 2nd line?

$\endgroup$
1
$\begingroup$

Spacy uses a hash function that assigns an integer to any Unicode string, it is not an index in vocabulary it just a random integer that is used internally for better efficiency. It is a hash function, so it means conflicts are indeed possible, but very unlikely and too rare to have a negative influence on the accuracy of the library, so the efficiency gains outweigh it.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.