Grouping domain specific words/phrases with same meaning

Question

I am looking at NLP methods to group together words/phrases which could have the same meaning. For example, in the sentence 'the table is broken' broken could be replaced by the following words/phrases and the sentence would still have the same meaning.

Broken: damaged, ruined, busted, unfit for purpose, missing a leg

I want to do this for texts that contain domain specific and colloquial jargon so existing NLP solutions may not be suitable?

My intention is to do this as a bridging step between named entity extraction and named entity linking.

Answer 1

I suggest you use word2vec for that task. Word2vec is an unsupervised algorithm that calculates N-dimension embeddings for the words in the corpus used for learning. Basically, it gives you a numerical representation in the form of an n-dimension array for each of the words you use in your inputs.

Once the model is built and the emeddings are available, to find similar words is as easy as calculate the similarity between arrays. This is done in word2vec either by looking for the most similar:

>>>result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man'])
>>>print("{}: {:.4f}".format(*result[0]))
queen: 0.7699

or directly comparing terms:

>>> result = word_vectors.similar_by_word("cat")
>>> print("{}: {:.4f}".format(*result[0]))
dog: 0.8798

Have a look here for more examples.

To answer this :

I want to do this for texts that contain domain specific and colloquial jargon so existing NLP solutions may not be suitable?

If you train the model with your own domain specific corpus, you should't have any problems; provided of course that it is long and rich enough.