Using BERT to extract a list of words and phrases from documents

Question

I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data.

For example my list of words and phrases look like,

Leadership
Microsoft Office
AWS
.
.
.
Python Programing Language

The result I am looking for is a matrix(3K x 3M) with binary values inside.

Doc #	Leadership	Microsoft Office	AWS	...	Python Programing Language
1	1	0	0		1
.	.	.	.		.
3M	0	1	1		1

1. Regex - This is the most straightforward solution comes my mind. However, this solutions is not robust and cannot capture different word/phrase forms. For example, people might write MS Office instead of Microsoft Office. Similarly, people might write Amazon Web Service rather than AWS.

2. Is there a solution to utilize a Large Language Model such as BERT?

3. If I create a labeled data using, for example, AWS Ground Truth, is there a way to utilize the results to build a model and extract the list of words/phrases?

Answer 1

It depends on the context imho: where does the list of words/phrases come from? Did some expert compile it thinking about every word carefully, or is it just intended as some rough indication of the actual context to be tagged?

Anyway from your description the output must use these particular phrases as columns, so it wouldn't make sense to use embeddings like BERT since the output would not really match the phrases themselves.

I don't see any point in a supervised model: what would be the target variable? All the phrases?? This would probably result in a very complex method for a very simple problem.

It might be disappointing but sometimes the simple answer is the appropriate one: direct search for these phrases, possibly after lemmatization to cover lexicographic variants.

Answer 2

You have a list of keyphrases that you need to extract from documents. You could use NER for this purpose. But looking at the size of the keyphrases (3000) it will be a difficult task because you would have to first annotate the keyphrases in the documents. After that you can train a NER model to make it learn to look for those phrases and extract them.

There are many NER models out there. Start with the SpaCy library first as it gives a simple but effective framework for NER. You can try multiple BERT based models using the SpaCy library. You can use any of the NLP models available on HuggingFace inside SpaCy for NER.