I'm trying to tokenize some sentences into phrases. For instance, given

I think you're cute and I want to know more about you

The tokens can be something like

I think you're cute


I want to know more about you

Similarly, given input

Today was great, but the weather could have been better.


Today was great


the weather could have been better

Can NLTK or similar packages achieve this?

Any advice appreciated.


1 Answer 1


Spacy can do this. Spacy's semantic parser is based on Language models trained on large corpus of text.

This parser can break sentence into lower level components such as words / phrases.

More details and examples :


Example with the first sentence from questions: https://explosion.ai/demos/displacy?text=I%20think%20you%27re%20cute%20and%20I%20want%20to%20know%20more%20about%20you&model=en_core_web_sm&cpu=0&cph=0

enter image description here

enter image description here

  • $\begingroup$ Thanks. I've had a look at semantic parsing, but it's not clear how to identify phrases. For instance, in my first sentence, a semantic parser can identify 'and' as CCONJ and 'think' and 'want' are connected by conj. It's not clear where the phrase boundaries actually are though. Is there a way to interpret the components to extract the boundaries? $\endgroup$
    – John M.
    Commented Jan 21, 2019 at 4:17
  • $\begingroup$ This is an example of parsing sub-trees. github.com/explosion/spacy/blob/master/examples/… I suggest that you try this example of few sentences and see if this works for your corpus. $\endgroup$ Commented Jan 21, 2019 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.