I intend to tokenize a number of job description texts. I have tried the standard tokenization using whitespace as the delimiter. However I noticed that there are some multi-word expressions that are splitted by whitespace, which may well cause accuracy problems in subsequent processing. So I want to get all the most interesting/informative collocations in these texts.
Is there any good packages for doing multi-word tokenization, regardless of specific programming language? For example, "He studies Information Technology" ===> "He" "studies" "Information Technology".
I've noticed NLTK (Python) has some related functionalities.
collocations Module: http://www.nltk.org/api/nltk.html#module-nltk.collocations
nltk.tokenize.mwe module: http://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.mwe
What the difference between these two?
The MWETokenizer class in the nltk.tokenize.mwe module seems working towards my objective. However MWETokenizer seems to require me to use its construction method and .add_mwe method to add multi-word expressions. Is there a way to use external multi-word expression lexicon to achieve this? If so, are there any multi-word lexicon?