Spacy Returns Nonidentical Results for Doc. Examples?

Question

I am getting started with Spacy, and I was just experimenting the given examples. It is weird though that when I run them over their containers using their experimental console it gives me different results than when I run them locally.

For example, when I run the following example from their console,

import spacy

nlp = spacy.load('en_core_web_md')  # make sure to use larger model!
tokens = nlp(u'dog cat banana')

for token1 in tokens:
    for token2 in tokens:
        print(token1.text, token2.text, token1.similarity(token2))

It gives me:

dog dog 1.0
dog cat 0.80168545
dog banana 0.24327646
cat dog 0.80168545
cat cat 1.0
cat banana 0.2815437
banana dog 0.24327646
banana cat 0.2815437
banana banana 1.0

But when I run the exact same thing locally, it gives me:

dog dog 1.0
dog cat 0.0
dog banana 0.0
cat dog 0.0
cat cat 1.0
cat banana -0.044681177
banana dog -7.828739e+17
banana cat -8.242222e+17
banana banana 1.0

To me the former makes sense, not my local result. And the problem persists for other examples as well. I am so confused! What is happening locally?

• Spacy: '2.0.12'
• Python 3.6.5 Anaconda, Inc.
• Mac OSx 10.13.5

I am not sure if it is relevant, but I was getting this error running it in Jupyter, so I exported the followings, as suggested in the git, in the terminal before launching the Jupyter:

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

Answer 1

After browsing the reported issues in spacy git page, it turns out that there is an issue mainly with certain version of packages specifically numpy. See this closed thread. As suggested, I did:

conda update --all

And everything works as expected with the following packages (even in ipython):

• Python 3.6.5 Anaconda, Inc.
• Spacy: 2.0.12
• Numpy: 1.15.2
• Ipython: 6.5.0
• Mac OSx 10.13.5

Happy NLP!

Answer 2

As I commented on your post - I think you have a problem with your environment. If you cannot solve that, you should file a bug with the spacy team on their GitHub.

I tested the same code and could only see a difference in results compared to the website when using language models of different sizes. If you installed the English model using the default command: python -m spacy download en, then you get the smallest model by default. You should be using the medium-sized model to produce their results. I compare the two below

The script

import spacy

# Small setup
nlp_small = spacy.load('en')
tokens_s = nlp_small(u'dog cat banana')

# Medium setup
nlp_medium = spacy.load('en_core_web_md')
tokens_m = nlp_medium(u'dog cat banana')

# large setup would use 'en_core_web_lg' ...


print('Results using small model:\n')
for token1 in tokens_s:
    for token2 in tokens_s:
        print(token1.text, token2.text, token1.similarity(token2))

print('\nResults using medium model:\n')
for token1 in tokens_m:
    for token2 in tokens_m:
        print(token1.text, token2.text, token1.similarity(token2))

The output

Results using small model:

dog dog 1.0
dog cat 0.53906965
dog banana 0.28761008
cat dog 0.53906965
cat cat 1.0
cat banana 0.4875216
banana dog 0.28761008
banana cat 0.4875216
banana banana 1.0

Results using medium model:

dog dog 1.0
dog cat 0.8016855
dog banana 0.24327648
cat dog 0.8016855
cat cat 1.0
cat banana 0.28154367
banana dog 0.24327648
banana cat 0.28154367
banana banana 1.0

The results are clearly better when using the medium sized model - and they match the results on the website.

---

Here is the spacy version as used within a conda environment to produce these results:

n1k31t4@n1k31t4~$ conda list | grep spacy
spacy                     2.0.12                    <pip>

And the Python version information:

n1k31t4@n1k31t4:~$ ipython
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

Although I ran the script on Linux (OP used MacOSX) and my Python version is slightly newer... I wouldn't expect these things to explain the difference in results.