Tokenisation with Spacy - how to get tokens to the left/right of token

Question

I am using Spacy for text tokenisation and getting stuck with it:

import spacy
nlp = spacy.load("en_core_web_sm")
mytext = "This is some sentence that spacy will not appreciate"
doc = nlp(mytext)

for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_, token.shape_, token.is_alpha, token.is_stop)

returns something that seems to me to say that tokenisation was succesful:

This this DET DT nsubj Xxxx True False 
is be VERB VBZ ROOT xx True True 
some some DET DT det xxxx True True 
sentence sentence NOUN NN attr xxxx True False 
that that ADP IN mark xxxx True True 
spacy spacy NOUN NN nsubj xxxx True False 
will will VERB MD aux xxxx True True 
not not ADV RB neg xxx True True 
appreciate appreciate VERB VB ccomp xxxx True False

but on the other hand

[token.text for token in doc[2].lefts]

returns an empty list. Is there a bug in lefts/rights?

Beginner at natural language processing, hope I am not falling into a conceptual trap. Using Spacy v'2.0.4'.

Answer 1

the token.lefts and token.rights attributes return a generator of the immediate children of the word, in the syntactic dependency parse. It does not just return the tokens on the left and right of the given token.

see : https://spacy.io/api/token#rights

If you want the adjacent tokens for a doc, you can do :

for i in range(len(doc))[1:-1]:
    print(doc[i-1], doc[i+1])

It will print the adjacent tokens for all tokens of the doc, starting at the 2nd token and finishing at the penultimate one.