2
$\begingroup$

Looking for a way to rank the tens and hundreds of named entities present in any document in order of their importance/relevance in the context.

Any thoughts ?

Thanks in advance!

$\endgroup$
0
$\begingroup$

An easy way would be to use TF-IDF (term frequency–inverse document frequency). It can help you find how much terms stand out in a document (by comparing with your entire corpus) and use it to rank your entities.

TfidfVectorizer from scikit-learn

Just note that the TfidfVectorizer is on a word level. So some processing will be needed if your entities can consist of more than one word.

Alternatively you could use a model that allows you to produce a heatmap of the words. Then you can use that heatmap to look up your NEs in that heatmap. This paper, A Structured Self-Attentive Sentence Embedding, could give you some ideas.

$\endgroup$
  • $\begingroup$ Thanks for your response, Simon! However, I was wondering if there is a better approach since tf-idf will not take into consideration the context of the document. $\endgroup$ – Neelam Aug 15 at 17:36
  • $\begingroup$ You are correct. I think you can achieve what you are looking for by using a self-attention model. I will link something in my answer. $\endgroup$ – Simon Larsson Aug 19 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.