1
$\begingroup$

I am searching for a python package that calculates the semantic similarity between words. I do not want to train a model (what most packages seem to offer) - the package should have been pre-trained on ideally thousands of natural language books and documents (e.g. on how often do words occur in close proximity to each other in the training material) and be simple to install/use. As for example in pseudo code below:

import XYZ

assessor = XYZ.loadPreTrainedModel("standard_text")
assessor.scoreWords("pilot", "airplane")  # returns 0.94 (I made up these numbers)
assessor.scoreWords("student", "university")  # returns 0.91
assessor.scoreWords("cat", "dog")  # returns 0.82 
assessor.scoreWords("cat", "airplane")  # returns 0.13
assessor.scoreWords("student", "apple")  # returns 0.25
...
$\endgroup$
3
$\begingroup$

The spaCy Python package might work for you. It allows you to easily "install" large pre-trained language models, and it provides a nice high-level interface for comparing word vectors.

To install spaCy:

pip install spacy

Then you need to download a language model. I believe these models are trained on Common Crawl, which is a massive dataset. You should choose one of the medium or large models, because the small models do not ship with word vectors.

python -m spacy download en_core_web_md

Using spacy models to compute word similarity is a breeze:

import spacy

# load the language model
nlp = spacy.load('en_core_web_md')

word1 = 'cat'
word2 = 'dog'

# convert the strings to spaCy Token objects
token1 = nlp(word1)[0]
token2 = nlp(word2)[0]

# compute word similarity
token1.similarity(token2)  # returns 0.80168

Here's an example that's more similar to the one in your question:

import spacy

nlp = spacy.load('en_core_web_md')
token = lambda word: nlp(word)[0]  # shortcut to convert string to spacy.Token
score_words = lambda w1, w2: token(w1).similarity(token(w2))

score_words("pilot", "airplane")      # 0.5998
score_words("student", "university")  # 0.7238
score_words("cat", "dog")             # 0.8017
score_words("cat", "airplane")        # 0.2654
score_words("student", "apple")       # 0.0928
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.