Looking for a way to rank the tens and hundreds of named entities present in any document in order of their importance/relevance in the context.
Any thoughts ?
Thanks in advance!
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
An easy way would be to use TF-IDF (term frequency–inverse document frequency). It can help you find how much terms stand out in a document (by comparing with your entire corpus) and use it to rank your entities.
Just note that the TfidfVectorizer is on a word level. So some processing will be needed if your entities can consist of more than one word.
Alternatively you could use a model that allows you to produce a heatmap of the words. Then you can use that heatmap to look up your NEs in that heatmap. This paper, A Structured Self-Attentive Sentence Embedding, could give you some ideas.