NLP: What are some popular packages for phrase tokenization?

Question

I'm trying to tokenize some sentences into phrases. For instance, given

I think you're cute and I want to know more about you

The tokens can be something like

I think you're cute

and

I want to know more about you

Similarly, given input

Today was great, but the weather could have been better.

Tokens:

Today was great

and

the weather could have been better

Can NLTK or similar packages achieve this?

Any advice appreciated.

Shamit Verma · Accepted Answer · 2019-01-20 15:26:24Z

0

Spacy can do this. Spacy's semantic parser is based on Language models trained on large corpus of text.

This parser can break sentence into lower level components such as words / phrases.

More details and examples :

answered Jan 20, 2019 at 15:26

Shamit Verma

2,2291 gold badge8 silver badges14 bronze badges

$\begingroup$ Thanks. I've had a look at semantic parsing, but it's not clear how to identify phrases. For instance, in my first sentence, a semantic parser can identify 'and' as CCONJ and 'think' and 'want' are connected by conj. It's not clear where the phrase boundaries actually are though. Is there a way to interpret the components to extract the boundaries? $\endgroup$
– John M.
Jan 21, 2019 at 4:17
$\begingroup$ This is an example of parsing sub-trees. github.com/explosion/spacy/blob/master/examples/… I suggest that you try this example of few sentences and see if this works for your corpus. $\endgroup$
– Shamit Verma
Jan 21, 2019 at 14:27

Add a comment |

1 Answer 1